[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:05 dist-test-slave-7491 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:05 dist-test-slave-7491 krb5kdc[395](info): set up 2 sockets
Sep 29 15:44:05 dist-test-slave-7491 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 15:44:06.993512 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:07 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:07.189782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35863
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:07.589789 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:07.590137 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:07.590298 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:07.598166 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:07.598320 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:07.598367 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:07.598506 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:07.598579 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:07.609768 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:07.611274 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:07.613446 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:07.629277 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:07.629355 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:07.629406 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:07.630224 408 server_base.cc:1034] running on GCE node
I20260929 15:44:07.631181 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:07.633450 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:07.634752 408 hybrid_clock.cc:648] HybridClock initialized: now 1790696647634656 us; error 79 us; skew 500 ppm
I20260929 15:44:07.635212 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:07 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:07.639635 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:07.641520 408 webserver.cc:460] Webserver started at http://127.0.97.62:33203/ using document root <none> and password file <none>
I20260929 15:44:07.642462 408 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:07.642582 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:07.642869 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:07.646296 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "0262d307f59643cbadb6388fe40b5836"
format_stamp: "Formatted at 2026-09-29 15:44:07 on dist-test-slave-7491"
server_key: "f6da2fc5f1d736a74c46ab9ea5d91ab3"
server_key_iv: "a5cb16df357b383f5130264ecdfac4fc"
server_key_version: "encryptionkey@0"
I20260929 15:44:07.647100 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "0262d307f59643cbadb6388fe40b5836"
format_stamp: "Formatted at 2026-09-29 15:44:07 on dist-test-slave-7491"
server_key: "f6da2fc5f1d736a74c46ab9ea5d91ab3"
server_key_iv: "a5cb16df357b383f5130264ecdfac4fc"
server_key_version: "encryptionkey@0"
I20260929 15:44:07.653684 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 15:44:07.657959 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:07.659617 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:44:07.659795 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "0262d307f59643cbadb6388fe40b5836"
format_stamp: "Formatted at 2026-09-29 15:44:07 on dist-test-slave-7491"
server_key: "f6da2fc5f1d736a74c46ab9ea5d91ab3"
server_key_iv: "a5cb16df357b383f5130264ecdfac4fc"
server_key_version: "encryptionkey@0"
I20260929 15:44:07.660058 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:07.680800 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:07.686990 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:07.687269 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:07.713573 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35863
I20260929 15:44:07.713613 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35863 every 8 connection(s)
I20260929 15:44:07.715230 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:07.720633 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:07.722508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 408
I20260929 15:44:07.723331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:07.725962 388 external_mini_cluster.cc:1428] Setting key dcf005efdbfd1c8d666c81b48ff33099
I20260929 15:44:07.736974 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836: Bootstrap starting.
I20260929 15:44:07.741197 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:07.742465 475 log.cc:824] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:07.746675 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836: No bootstrap required, opened a new log
I20260929 15:44:07.757756 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0262d307f59643cbadb6388fe40b5836" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35863 } }
I20260929 15:44:07.758430 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:07.758605 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0262d307f59643cbadb6388fe40b5836, State: Initialized, Role: FOLLOWER
I20260929 15:44:07.759454 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0262d307f59643cbadb6388fe40b5836" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35863 } }
I20260929 15:44:07.759764 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:07.759948 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:07.760140 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 15:44:07 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:07.763013 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0262d307f59643cbadb6388fe40b5836" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35863 } }
I20260929 15:44:07.763686 475 leader_election.cc:302] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0262d307f59643cbadb6388fe40b5836; no voters:
I20260929 15:44:07.764482 475 leader_election.cc:288] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:07.765002 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:07.767841 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:07.767940 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [term 1 LEADER]: Becoming Leader. State: Replica: 0262d307f59643cbadb6388fe40b5836, State: Running, Role: LEADER
I20260929 15:44:07.768769 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0262d307f59643cbadb6388fe40b5836" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35863 } }
I20260929 15:44:07.780398 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0262d307f59643cbadb6388fe40b5836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0262d307f59643cbadb6388fe40b5836" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35863 } } }
I20260929 15:44:07.780939 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:07.781682 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0262d307f59643cbadb6388fe40b5836. Latest consensus state: current_term: 1 leader_uuid: "0262d307f59643cbadb6388fe40b5836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0262d307f59643cbadb6388fe40b5836" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35863 } } }
I20260929 15:44:07.782608 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:07.787201 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:07.731617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.97.62:35863)
0929 15:44:07.732933 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:07.732957 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:07.734474 (+ 1517us) server_negotiation.cc:407] Connection header received
0929 15:44:07.745028 (+ 10554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:07.745063 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:07.745663 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:07.746198 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:07.750388 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:07.753646 (+ 3258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:07.756221 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:07.757113 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:07.757384 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:07.764951 (+ 7567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:07.764967 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:07.764999 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:07.765089 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:07.770668 (+ 5579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:07.772462 (+ 1794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:07.772477 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:07.772490 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:07.772811 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:07.773647 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:07.773654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:07.773660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:07.774997 (+ 1337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:07.779626 (+ 4629us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:07.780291 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 15:44:07.780999 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":673,"threads_started":1}
W20260929 15:44:07.794193 495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:44:07.794509 495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:44:07.794695 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:07.797175 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:07.810097 491 catalog_manager.cc:1347] Generated new cluster ID: 3941cd7d6d4f457789a6541a6f1f793d
I20260929 15:44:07.810379 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:07.823264 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:07.825722 491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:07.836469 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0262d307f59643cbadb6388fe40b5836: Generated new TSK 0
I20260929 15:44:07.837747 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:07.889355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35863
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:08.287318 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:08.287612 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:08.287709 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:08.295753 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:08.295912 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:08.296042 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:08.306802 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:08.308128 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:08.309576 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:08.323920 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:08.323959 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:08.324465 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:08.326120 501 server_base.cc:1034] running on GCE node
I20260929 15:44:08.326992 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:08.328096 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:08.329407 501 hybrid_clock.cc:648] HybridClock initialized: now 1790696648329347 us; error 40 us; skew 500 ppm
I20260929 15:44:08.329778 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:08 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:08.333462 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:08.335170 501 webserver.cc:460] Webserver started at http://127.0.97.1:39859/ using document root <none> and password file <none>
I20260929 15:44:08.335919 501 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:08.336071 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:08.336473 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:08.340477 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "7d7174529e9e4be8a4f824992383a996"
format_stamp: "Formatted at 2026-09-29 15:44:08 on dist-test-slave-7491"
server_key: "e68f68b1d5541d15476cc852953e2e39"
server_key_iv: "2ecdca3f867f312d789898fe20399afe"
server_key_version: "encryptionkey@0"
I20260929 15:44:08.341522 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7d7174529e9e4be8a4f824992383a996"
format_stamp: "Formatted at 2026-09-29 15:44:08 on dist-test-slave-7491"
server_key: "e68f68b1d5541d15476cc852953e2e39"
server_key_iv: "2ecdca3f867f312d789898fe20399afe"
server_key_version: "encryptionkey@0"
I20260929 15:44:08.347797 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 15:44:08.353610 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:08.355432 501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 15:44:08.355671 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "7d7174529e9e4be8a4f824992383a996"
format_stamp: "Formatted at 2026-09-29 15:44:08 on dist-test-slave-7491"
server_key: "e68f68b1d5541d15476cc852953e2e39"
server_key_iv: "2ecdca3f867f312d789898fe20399afe"
server_key_version: "encryptionkey@0"
I20260929 15:44:08.355935 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:08.373541 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:08.378724 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:08.379045 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:08.380069 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:08.381546 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:08.381652 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:08.381804 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:08.381881 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:08.426496 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36521
I20260929 15:44:08.426992 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36521 every 8 connection(s)
I20260929 15:44:08.428131 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:08.433624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 501
I20260929 15:44:08.433851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:08.434468 388 external_mini_cluster.cc:1428] Setting key cca5429bff7e373f6d46e278bf140413
Sep 29 15:44:08 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:44:08.452440 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:08.431489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54135 (local address 127.0.97.62:35863)
0929 15:44:08.431963 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:08.431967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:08.433322 (+ 1355us) server_negotiation.cc:407] Connection header received
0929 15:44:08.435534 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:08.435539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:08.435636 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:08.435770 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:08.437812 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:08.438595 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:08.440443 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:08.440895 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:08.441056 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:08.445275 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:08.445281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:08.445285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:08.445320 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:08.449101 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:08.449971 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:08.449998 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:08.450000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:08.450120 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:08.450593 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:08.450607 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:08.450609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:08.450883 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:08.451041 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:08.451788 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 15:44:08.452038 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260929 15:44:08.453516 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:08.432252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35863 (local address 127.0.97.1:54135)
0929 15:44:08.433130 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:08.433176 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:08.435197 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:08.436028 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:08.436038 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:08.436811 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:08.437585 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:08.437607 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:08.438789 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:08.438802 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:08.440227 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:08.440257 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:08.441174 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:08.441183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:08.441546 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:08.442449 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:08.442476 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:08.444990 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:08.449314 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:08.449335 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:08.449353 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:08.449825 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:08.450250 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:08.450256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:08.450264 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:08.450468 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:08.451017 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:08.451026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:08.451406 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 15:44:08.451677 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 15:44:08.452269 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":354,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:44:08.455305 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35863
I20260929 15:44:08.455649 629 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:08.456501 629 heartbeater.cc:511] Master 127.0.97.62:35863 requested a full tablet report, sending...
I20260929 15:44:08.458981 440 ts_manager.cc:194] Registered new tserver with Master: 7d7174529e9e4be8a4f824992383a996 (127.0.97.1:36521)
I20260929 15:44:08.461886 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:08.481667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35863
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:08.860879 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:08.861263 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:08.861399 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:08.869706 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:08.869848 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:08.869973 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:08.880748 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:08.882385 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:08.884347 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:08.900869 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:08.900987 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:08.900911 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:08.901760 635 server_base.cc:1034] running on GCE node
I20260929 15:44:08.902474 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:08.903565 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:08.904799 635 hybrid_clock.cc:648] HybridClock initialized: now 1790696648904751 us; error 40 us; skew 500 ppm
I20260929 15:44:08.905143 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:08 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:08.908601 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:08.910782 635 webserver.cc:460] Webserver started at http://127.0.97.2:41535/ using document root <none> and password file <none>
I20260929 15:44:08.911936 635 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:08.912129 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:08.912744 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:08.916702 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "294f2bd6ecec4f178de67f0c8d27d098"
format_stamp: "Formatted at 2026-09-29 15:44:08 on dist-test-slave-7491"
server_key: "05d14b9795511e3b84567073602a79c4"
server_key_iv: "aa9bec9655c768e709a8ceb6058d2c70"
server_key_version: "encryptionkey@0"
I20260929 15:44:08.917662 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "294f2bd6ecec4f178de67f0c8d27d098"
format_stamp: "Formatted at 2026-09-29 15:44:08 on dist-test-slave-7491"
server_key: "05d14b9795511e3b84567073602a79c4"
server_key_iv: "aa9bec9655c768e709a8ceb6058d2c70"
server_key_version: "encryptionkey@0"
I20260929 15:44:08.923743 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:44:08.927944 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:08.929682 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:44:08.929919 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "294f2bd6ecec4f178de67f0c8d27d098"
format_stamp: "Formatted at 2026-09-29 15:44:08 on dist-test-slave-7491"
server_key: "05d14b9795511e3b84567073602a79c4"
server_key_iv: "aa9bec9655c768e709a8ceb6058d2c70"
server_key_version: "encryptionkey@0"
I20260929 15:44:08.930150 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:08.946427 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:08.951620 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:08.951972 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:08.953501 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:08.954945 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:08.955019 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:08.955175 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:08.955240 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:08.997583 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46281
I20260929 15:44:08.997627 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46281 every 8 connection(s)
I20260929 15:44:08.998944 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:09.003394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 635
I20260929 15:44:09.003623 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:09.004211 388 external_mini_cluster.cc:1428] Setting key 2ffb61bdbf7b3411ae7c5a594a0053ee
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:44:09.027805 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.002135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51361 (local address 127.0.97.62:35863)
0929 15:44:09.002628 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:09.002633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:09.004264 (+ 1631us) server_negotiation.cc:407] Connection header received
0929 15:44:09.006682 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:09.006686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:09.006762 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:09.006871 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:09.010363 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.011116 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.013637 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.014082 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.014231 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.019106 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:09.019112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:09.019115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:09.019152 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:09.022769 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.023820 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.023827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.023829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.024035 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.024967 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.024972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.024974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.025488 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:09.025857 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:09.026940 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0929 15:44:09.027245 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":281,"threads_started":1}
I20260929 15:44:09.028595 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.002798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35863 (local address 127.0.97.2:51361)
0929 15:44:09.004002 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:09.004074 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:09.006304 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:09.007214 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:09.007228 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:09.009012 (+ 1784us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:09.010056 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:09.010094 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.011319 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.011328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:09.013450 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:09.013465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.014438 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.014449 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:09.014716 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.015585 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:09.015611 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:09.018777 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:09.022964 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:09.022982 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:09.022999 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:09.023566 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:09.024281 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:09.024293 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:09.024297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:09.024790 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:09.025773 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:09.025787 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:09.026328 (+ 541us) client_negotiation.cc:769] Sending connection context
0929 15:44:09.026742 (+ 414us) client_negotiation.cc:240] Negotiation successful
0929 15:44:09.027277 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":439,"threads_started":1}
I20260929 15:44:09.030601 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35863
I20260929 15:44:09.030982 763 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:09.031792 763 heartbeater.cc:511] Master 127.0.97.62:35863 requested a full tablet report, sending...
I20260929 15:44:09.033659 440 ts_manager.cc:194] Registered new tserver with Master: 294f2bd6ecec4f178de67f0c8d27d098 (127.0.97.2:46281)
I20260929 15:44:09.035027 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:09.052613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35863
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:44:09.466943 629 heartbeater.cc:503] Master 127.0.97.62:35863 was elected leader, sending a full tablet report...
W20260929 15:44:09.472342 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:09.472813 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:09.473016 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:09.482769 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:09.482972 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:09.483179 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:09.495671 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:09.497977 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:09.499637 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:09.516287 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:09.516204 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:09.516460 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:09.517280 769 server_base.cc:1034] running on GCE node
I20260929 15:44:09.518464 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:09.519868 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:09.521203 769 hybrid_clock.cc:648] HybridClock initialized: now 1790696649521152 us; error 47 us; skew 500 ppm
I20260929 15:44:09.521617 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:09.525619 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:09.527488 769 webserver.cc:460] Webserver started at http://127.0.97.3:46415/ using document root <none> and password file <none>
I20260929 15:44:09.528347 769 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:09.528499 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:09.528861 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:09.532493 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "7187f088d3664c639603ddee5aa05d67"
format_stamp: "Formatted at 2026-09-29 15:44:09 on dist-test-slave-7491"
server_key: "75d46edc2003790d7ef76129de1e56dd"
server_key_iv: "fb65afd156faea98e316d70b632f4a25"
server_key_version: "encryptionkey@0"
I20260929 15:44:09.533555 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7187f088d3664c639603ddee5aa05d67"
format_stamp: "Formatted at 2026-09-29 15:44:09 on dist-test-slave-7491"
server_key: "75d46edc2003790d7ef76129de1e56dd"
server_key_iv: "fb65afd156faea98e316d70b632f4a25"
server_key_version: "encryptionkey@0"
I20260929 15:44:09.541977 769 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260929 15:44:09.546618 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:09.548589 769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:44:09.548918 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "7187f088d3664c639603ddee5aa05d67"
format_stamp: "Formatted at 2026-09-29 15:44:09 on dist-test-slave-7491"
server_key: "75d46edc2003790d7ef76129de1e56dd"
server_key_iv: "fb65afd156faea98e316d70b632f4a25"
server_key_version: "encryptionkey@0"
I20260929 15:44:09.549163 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:09.574270 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:09.580319 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:09.580677 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:09.581816 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:09.583480 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:09.583568 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:09.583709 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:09.583765 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:09.627580 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37593
I20260929 15:44:09.627615 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37593 every 8 connection(s)
I20260929 15:44:09.629012 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:09.636492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 769
I20260929 15:44:09.636718 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:09.637252 388 external_mini_cluster.cc:1428] Setting key 5ffe44f60a29532754dd4b03f4347cf7
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:09.655021 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.632511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50601 (local address 127.0.97.62:35863)
0929 15:44:09.633231 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:09.633258 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:09.633937 (+ 679us) server_negotiation.cc:407] Connection header received
0929 15:44:09.636831 (+ 2894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:09.636840 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:09.636927 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:09.637027 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:09.639582 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.640473 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.642612 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.643025 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.643163 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.647109 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:09.647115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:09.647118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:09.647152 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:09.650385 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.651359 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.651365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.651367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.651536 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.652290 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.652294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.652297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.652610 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:09.652752 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:09.654395 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0929 15:44:09.654677 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":341,"threads_started":1}
I20260929 15:44:09.656549 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.632876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35863 (local address 127.0.97.3:50601)
0929 15:44:09.633721 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:09.633791 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:09.636423 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:09.637335 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:09.637351 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:09.638205 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:09.639239 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:09.639268 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.640656 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.640662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:09.642447 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:09.642462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.643273 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.643283 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:09.643555 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.644370 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:09.644396 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:09.646845 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:09.650620 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:09.650628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:09.650644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:09.651154 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:09.651696 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:09.651700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:09.651702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:09.651905 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:09.652837 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:09.652851 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:09.653584 (+ 733us) client_negotiation.cc:769] Sending connection context
0929 15:44:09.654051 (+ 467us) client_negotiation.cc:240] Negotiation successful
0929 15:44:09.654800 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":361,"threads_started":1}
I20260929 15:44:09.658944 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35863
I20260929 15:44:09.659353 897 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:09.660319 897 heartbeater.cc:511] Master 127.0.97.62:35863 requested a full tablet report, sending...
I20260929 15:44:09.662143 440 ts_manager.cc:194] Registered new tserver with Master: 7187f088d3664c639603ddee5aa05d67 (127.0.97.3:37593)
I20260929 15:44:09.663044 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50601
I20260929 15:44:09.669915 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:09.726017 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.704642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.0.97.1:36521)
0929 15:44:09.705498 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:09.705510 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:09.705552 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:44:09.705638 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:09.705653 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:09.706109 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:09.706292 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:09.707789 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.709241 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.711097 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.711701 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.711824 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.718005 (+ 6181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:09.718032 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:09.718060 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:09.718131 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:09.722774 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.723686 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.723693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.723695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.723828 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.724429 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.724433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.724435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.724843 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:09.725086 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:09.725499 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 15:44:09.725721 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":424,"threads_started":1}
W20260929 15:44:09.727413 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46328
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:09.780696 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.761712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.97.1:36521)
0929 15:44:09.761962 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:09.761968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:09.762229 (+ 261us) server_negotiation.cc:407] Connection header received
0929 15:44:09.762326 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:09.762330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:09.762396 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:09.762488 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:09.763901 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.764741 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.766019 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.766504 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.766613 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.770916 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:09.770922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:09.770925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:09.770964 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:09.776378 (+ 5414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.777531 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.777540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.777544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.777819 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.778566 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.778570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.778573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.778909 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:09.779195 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:09.779827 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 15:44:09.780128 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:44:09.782682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 501
I20260929 15:44:09.799885 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.783787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.97.1:36521)
0929 15:44:09.784061 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:09.784069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:09.784605 (+ 536us) server_negotiation.cc:407] Connection header received
0929 15:44:09.784788 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:09.784792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:09.784864 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:09.784960 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:09.786174 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.786922 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.788326 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.788759 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.788871 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.792309 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:09.792316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:09.792321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:09.792369 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:09.795693 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.796839 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.796845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.796848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.797102 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.797737 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.797741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.797743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.798065 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:09.798323 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:09.798750 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 15:44:09.798967 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 15:44:09.800843 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:09.891459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 501
I20260929 15:44:09.911777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 635
Sep 29 15:44:09 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:09.932669 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:09.913147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.97.2:46281)
0929 15:44:09.913839 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:09.913854 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:09.913887 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:44:09.913966 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:09.913974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:09.914294 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:09.914463 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:09.916287 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.917509 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.919085 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:09.919775 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:09.919922 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:09.924860 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:09.924879 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:09.924898 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:09.924958 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:09.928410 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.929322 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.929331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.929333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.929491 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:09.930250 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:09.930257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:09.930260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:09.930865 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:09.931216 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:09.931793 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 15:44:09.932089 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":381,"threads_started":1}
I20260929 15:44:09.934554 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:10.032460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 635
I20260929 15:44:10.058063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 769
Sep 29 15:44:10 dist-test-slave-7491 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:10.079066 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:10.059766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.97.3:37593)
0929 15:44:10.060534 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:10.060542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:10.060579 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:44:10.060648 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:10.060655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:10.060911 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:10.061067 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:10.062457 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:10.063609 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:10.065364 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:10.065966 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:10.066132 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:10.071271 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:10.071290 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:10.071307 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:10.071357 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:10.075378 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:10.076544 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:10.076557 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:10.076561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:10.076794 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:10.077377 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:10.077382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:10.077384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:10.077864 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:10.078292 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:10.078504 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 15:44:10.078741 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":425,"threads_started":1}
I20260929 15:44:10.080767 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:10.169363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 769
I20260929 15:44:10.187949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 408
I20260929 15:44:10.189270 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:10.306905 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 408
2026-09-29T15:44:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4489 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:10 dist-test-slave-7491 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:10 dist-test-slave-7491 krb5kdc[919](info): set up 2 sockets
Sep 29 15:44:10 dist-test-slave-7491 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260929 15:44:11.432746 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:11 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:11.645773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44113
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:12.044600 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:12.045029 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:12.045202 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:12.054701 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:12.054961 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:12.055101 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:12.055217 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:12.055303 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:12.069617 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:12.071600 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:12.073192 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:12.086808 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:12.086829 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:12.086943 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:12.087296 932 server_base.cc:1034] running on GCE node
I20260929 15:44:12.088900 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:12.090652 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:12.091952 932 hybrid_clock.cc:648] HybridClock initialized: now 1790696652091879 us; error 68 us; skew 500 ppm
I20260929 15:44:12.092657 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:12 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:12.101321 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:12.103297 932 webserver.cc:460] Webserver started at http://127.0.97.62:33991/ using document root <none> and password file <none>
I20260929 15:44:12.104151 932 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:12.104398 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:12.104758 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:12.108304 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "2bc02f9069be406eabb328875a45a3e3"
format_stamp: "Formatted at 2026-09-29 15:44:12 on dist-test-slave-7491"
server_key: "904f59caff06b744097579f3cf183151"
server_key_iv: "11e73a37a5d06977bd5d3548b0b92102"
server_key_version: "encryptionkey@0"
I20260929 15:44:12.109205 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "2bc02f9069be406eabb328875a45a3e3"
format_stamp: "Formatted at 2026-09-29 15:44:12 on dist-test-slave-7491"
server_key: "904f59caff06b744097579f3cf183151"
server_key_iv: "11e73a37a5d06977bd5d3548b0b92102"
server_key_version: "encryptionkey@0"
I20260929 15:44:12.118382 932 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260929 15:44:12.123332 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:12.125353 932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:44:12.125705 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "2bc02f9069be406eabb328875a45a3e3"
format_stamp: "Formatted at 2026-09-29 15:44:12 on dist-test-slave-7491"
server_key: "904f59caff06b744097579f3cf183151"
server_key_iv: "11e73a37a5d06977bd5d3548b0b92102"
server_key_version: "encryptionkey@0"
I20260929 15:44:12.125949 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:12.139601 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:12.145326 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:12.145887 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:12.171125 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44113
I20260929 15:44:12.171125 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44113 every 8 connection(s)
I20260929 15:44:12.173167 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:12.178051 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:12.180496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 932
I20260929 15:44:12.180811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:12.181702 388 external_mini_cluster.cc:1428] Setting key ba6573e0d52c9d6e235f53d9e5321b7b
I20260929 15:44:12.193542 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3: Bootstrap starting.
I20260929 15:44:12.197964 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:12.199450 999 log.cc:824] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:44:12 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:12.203763 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3: No bootstrap required, opened a new log
I20260929 15:44:12.214406 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:12.184766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.97.62:44113)
0929 15:44:12.185684 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:12.185698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:12.185752 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:44:12.187574 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:12.187599 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:12.188056 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:12.188513 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:12.190641 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:12.192684 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:12.194537 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:12.195477 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:12.195648 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:12.201278 (+ 5630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:12.201292 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:12.201318 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:12.201376 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:12.205815 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:12.206754 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:12.206766 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:12.206778 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:12.207056 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:12.207527 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:12.207534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:12.207537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:12.208467 (+ 930us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:12.208924 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:12.209652 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 15:44:12.210397 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":428,"threads_started":1}
I20260929 15:44:12.218858 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc02f9069be406eabb328875a45a3e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260929 15:44:12.219451 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:12.219587 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc02f9069be406eabb328875a45a3e3, State: Initialized, Role: FOLLOWER
I20260929 15:44:12.220468 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc02f9069be406eabb328875a45a3e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260929 15:44:12.221000 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:12.221319 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:12.221632 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:12.224385 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc02f9069be406eabb328875a45a3e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260929 15:44:12.225067 999 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc02f9069be406eabb328875a45a3e3; no voters:
I20260929 15:44:12.225762 999 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:12.225934 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:12.226742 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc02f9069be406eabb328875a45a3e3, State: Running, Role: LEADER
I20260929 15:44:12.227342 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc02f9069be406eabb328875a45a3e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } }
I20260929 15:44:12.228212 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:12.231168 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc02f9069be406eabb328875a45a3e3. Latest consensus state: current_term: 1 leader_uuid: "2bc02f9069be406eabb328875a45a3e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc02f9069be406eabb328875a45a3e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } } }
I20260929 15:44:12.231737 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:12.232565 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc02f9069be406eabb328875a45a3e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc02f9069be406eabb328875a45a3e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44113 } } }
I20260929 15:44:12.233006 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:12.236495 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:12.242175 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:12.253273 1012 catalog_manager.cc:1347] Generated new cluster ID: a1bbd556f3364079b799bd829a6dbf23
I20260929 15:44:12.253414 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:12.276298 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:12.278259 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:12.286653 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc02f9069be406eabb328875a45a3e3: Generated new TSK 0
I20260929 15:44:12.287803 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:12.341583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44113
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:12.746841 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:12.747241 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:12.747414 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:12.756649 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:12.756898 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:12.757112 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:12.768631 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:12.770519 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:12.773025 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:12.790338 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:12.790376 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:12.790370 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:12.791653 1025 server_base.cc:1034] running on GCE node
I20260929 15:44:12.792773 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:12.794324 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:12.795668 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790696652795618 us; error 51 us; skew 500 ppm
I20260929 15:44:12.796152 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:12 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:12.800679 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:12.802556 1025 webserver.cc:460] Webserver started at http://127.0.97.1:41229/ using document root <none> and password file <none>
I20260929 15:44:12.803365 1025 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:12.803530 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:12.803886 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:12.807276 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ea751778d284bca8441f115d9853c9f"
format_stamp: "Formatted at 2026-09-29 15:44:12 on dist-test-slave-7491"
server_key: "c53d72ebc86e4497290f7cd16de7ca3d"
server_key_iv: "8680acdef5e1015bbec22a75a3d30c11"
server_key_version: "encryptionkey@0"
I20260929 15:44:12.808319 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ea751778d284bca8441f115d9853c9f"
format_stamp: "Formatted at 2026-09-29 15:44:12 on dist-test-slave-7491"
server_key: "c53d72ebc86e4497290f7cd16de7ca3d"
server_key_iv: "8680acdef5e1015bbec22a75a3d30c11"
server_key_version: "encryptionkey@0"
I20260929 15:44:12.815075 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:12.820984 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:12.823043 1025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 15:44:12.823319 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "2ea751778d284bca8441f115d9853c9f"
format_stamp: "Formatted at 2026-09-29 15:44:12 on dist-test-slave-7491"
server_key: "c53d72ebc86e4497290f7cd16de7ca3d"
server_key_iv: "8680acdef5e1015bbec22a75a3d30c11"
server_key_version: "encryptionkey@0"
I20260929 15:44:12.823593 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:12.851843 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:12.858623 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:12.859025 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:12.860647 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:12.862572 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:12.862718 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:12.862900 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:12.862978 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:12.904911 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46223
I20260929 15:44:12.904928 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46223 every 8 connection(s)
I20260929 15:44:12.906338 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:12.907181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1025
I20260929 15:44:12.907473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:12.908306 388 external_mini_cluster.cc:1428] Setting key ef1758c1e2446ebd032556fb47cde017
Sep 29 15:44:12 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:44:12.938491 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:12.910845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57163 (local address 127.0.97.62:44113)
0929 15:44:12.911497 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:12.911504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:12.913376 (+ 1872us) server_negotiation.cc:407] Connection header received
0929 15:44:12.915882 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:12.915892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:12.916326 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:12.916691 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:12.919118 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:12.919909 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:12.921906 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:12.922340 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:12.922493 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:12.927921 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:12.927934 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:12.927942 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:12.928010 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:12.932534 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:12.934446 (+ 1912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:12.934452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:12.934455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:12.934665 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:12.935529 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:12.935536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:12.935539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:12.936108 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:12.936426 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:12.937823 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0929 15:44:12.938167 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":321,"threads_started":1}
I20260929 15:44:12.940096 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:12.911791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44113 (local address 127.0.97.1:57163)
0929 15:44:12.913092 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:12.913157 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:12.915435 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:12.916969 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:12.916980 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:12.917667 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:12.918735 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:12.918768 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:12.920246 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:12.920258 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:12.921677 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:12.921696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:12.922636 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:12.922644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:12.922919 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:12.923860 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:12.923896 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:12.927428 (+ 3532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:12.932889 (+ 5461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:12.932906 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:12.932932 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:12.934075 (+ 1143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:12.934819 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:12.934828 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:12.934836 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:12.935216 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:12.936406 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:12.936421 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:12.937165 (+ 744us) client_negotiation.cc:769] Sending connection context
0929 15:44:12.937639 (+ 474us) client_negotiation.cc:240] Negotiation successful
0929 15:44:12.938478 (+ 839us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":580,"threads_started":1}
I20260929 15:44:12.942783 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44113
I20260929 15:44:12.943370 1153 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:12.944448 1153 heartbeater.cc:511] Master 127.0.97.62:44113 requested a full tablet report, sending...
I20260929 15:44:12.947945 964 ts_manager.cc:194] Registered new tserver with Master: 2ea751778d284bca8441f115d9853c9f (127.0.97.1:46223)
I20260929 15:44:12.950661 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:12.964819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44113
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:13.353180 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:13.353585 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:13.353763 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:13.361703 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:13.361925 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:13.362129 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:13.373643 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:13.375272 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:13.376906 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:13.392504 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:13.392504 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:13.392496 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:13.393514 1159 server_base.cc:1034] running on GCE node
I20260929 15:44:13.394331 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:13.395601 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:13.396936 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790696653396864 us; error 73 us; skew 500 ppm
I20260929 15:44:13.397373 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:13 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:13.402514 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:13.404990 1159 webserver.cc:460] Webserver started at http://127.0.97.2:44425/ using document root <none> and password file <none>
I20260929 15:44:13.406126 1159 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:13.406309 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:13.406734 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:13.411252 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "2abafdfa495c4115aa370710dbf7d225"
format_stamp: "Formatted at 2026-09-29 15:44:13 on dist-test-slave-7491"
server_key: "dfd52bd36f27118fb8044c840980a050"
server_key_iv: "145c448634e1b415660c0e4da3d4fc49"
server_key_version: "encryptionkey@0"
I20260929 15:44:13.412451 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2abafdfa495c4115aa370710dbf7d225"
format_stamp: "Formatted at 2026-09-29 15:44:13 on dist-test-slave-7491"
server_key: "dfd52bd36f27118fb8044c840980a050"
server_key_iv: "145c448634e1b415660c0e4da3d4fc49"
server_key_version: "encryptionkey@0"
I20260929 15:44:13.418982 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:13.423903 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:13.425909 1159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:13.426182 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "2abafdfa495c4115aa370710dbf7d225"
format_stamp: "Formatted at 2026-09-29 15:44:13 on dist-test-slave-7491"
server_key: "dfd52bd36f27118fb8044c840980a050"
server_key_iv: "145c448634e1b415660c0e4da3d4fc49"
server_key_version: "encryptionkey@0"
I20260929 15:44:13.426463 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:13.458374 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:13.464800 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:13.465276 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:13.466784 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:13.468888 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:13.469009 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:13.469199 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:13.469285 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:13.510061 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35131
I20260929 15:44:13.510104 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35131 every 8 connection(s)
I20260929 15:44:13.511463 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:13.519479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1159
I20260929 15:44:13.519721 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:13.520151 388 external_mini_cluster.cc:1428] Setting key f5ff01f9450d3ba5922e66ae23aa8a7a
Sep 29 15:44:13 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:13.538070 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:13.515161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42427 (local address 127.0.97.62:44113)
0929 15:44:13.515700 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:13.515707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:13.516986 (+ 1279us) server_negotiation.cc:407] Connection header received
0929 15:44:13.519015 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:13.519020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:13.519101 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:13.519281 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:13.521332 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:13.522479 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:13.524540 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:13.525004 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:13.525161 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:13.529636 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:13.529642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:13.529645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:13.529682 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:13.533241 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:13.534570 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:13.534576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:13.534580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:13.534757 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:13.535355 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:13.535361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:13.535365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:13.535844 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:13.536062 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:13.537186 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0929 15:44:13.537500 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":248,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:44:13.539885 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:13.515824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44113 (local address 127.0.97.2:42427)
0929 15:44:13.516787 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:13.516839 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:13.518654 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:13.519602 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:13.519613 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:13.520447 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:13.521127 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:13.521148 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:13.522687 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:13.522692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:13.524102 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:13.524126 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:13.525279 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:13.525287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:13.525556 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:13.526408 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:13.526435 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:13.529262 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:13.533601 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:13.533615 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:13.533638 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:13.534381 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:13.534901 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:13.534907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:13.534912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:13.535205 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:13.535991 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:13.536004 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:13.536649 (+ 645us) client_negotiation.cc:769] Sending connection context
0929 15:44:13.537077 (+ 428us) client_negotiation.cc:240] Negotiation successful
0929 15:44:13.538116 (+ 1039us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":452,"threads_started":1}
I20260929 15:44:13.542162 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:44:13.542586 1287 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:13.543454 1287 heartbeater.cc:511] Master 127.0.97.62:44113 requested a full tablet report, sending...
I20260929 15:44:13.545576 964 ts_manager.cc:194] Registered new tserver with Master: 2abafdfa495c4115aa370710dbf7d225 (127.0.97.2:35131)
I20260929 15:44:13.546927 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:13.568995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44113
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:13.946285 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:13.946623 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:13.946723 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:13.954562 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:13.954710 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:13.954838 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:13.956068 1153 heartbeater.cc:503] Master 127.0.97.62:44113 was elected leader, sending a full tablet report...
I20260929 15:44:13.965965 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:13.967514 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:13.969161 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:13.986119 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:13.986073 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:13.986153 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:13.987074 1293 server_base.cc:1034] running on GCE node
I20260929 15:44:13.987823 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:13.989184 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:13.990571 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790696653990453 us; error 81 us; skew 500 ppm
I20260929 15:44:13.991184 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:13 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:13.995908 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:13.998665 1293 webserver.cc:460] Webserver started at http://127.0.97.3:43719/ using document root <none> and password file <none>
I20260929 15:44:13.999739 1293 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:13.999873 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:14.000438 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:14.004642 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "e92c9024b5974f4b8f890d4aae0abff0"
format_stamp: "Formatted at 2026-09-29 15:44:13 on dist-test-slave-7491"
server_key: "e515d12f7e6cf9019323a43845a144ad"
server_key_iv: "b6cb5cc5d0a4f785d4857f69b8d8f6cc"
server_key_version: "encryptionkey@0"
I20260929 15:44:14.005877 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e92c9024b5974f4b8f890d4aae0abff0"
format_stamp: "Formatted at 2026-09-29 15:44:13 on dist-test-slave-7491"
server_key: "e515d12f7e6cf9019323a43845a144ad"
server_key_iv: "b6cb5cc5d0a4f785d4857f69b8d8f6cc"
server_key_version: "encryptionkey@0"
I20260929 15:44:14.013432 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 15:44:14.018638 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:14.020541 1293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:44:14.020767 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "e92c9024b5974f4b8f890d4aae0abff0"
format_stamp: "Formatted at 2026-09-29 15:44:13 on dist-test-slave-7491"
server_key: "e515d12f7e6cf9019323a43845a144ad"
server_key_iv: "b6cb5cc5d0a4f785d4857f69b8d8f6cc"
server_key_version: "encryptionkey@0"
I20260929 15:44:14.021023 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:14.044800 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:14.051661 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:14.052093 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:14.053695 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:14.055573 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:14.055667 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:14.055816 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:14.055892 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:14.098604 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33529
I20260929 15:44:14.098655 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33529 every 8 connection(s)
I20260929 15:44:14.100543 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:14.102103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1293
I20260929 15:44:14.102336 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:14.102914 388 external_mini_cluster.cc:1428] Setting key cf3ffb055446d32bb9098e126f8b6e87
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:14.128080 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.104705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38325 (local address 127.0.97.62:44113)
0929 15:44:14.105386 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.105394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.106938 (+ 1544us) server_negotiation.cc:407] Connection header received
0929 15:44:14.109850 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.109854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.109970 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.110100 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.112358 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.113174 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.115553 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.116023 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.116149 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.120802 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.120809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.120814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.120860 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.124425 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.125373 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.125386 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.125390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.125553 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.126035 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.126047 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.126050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.126344 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.126514 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.127318 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.127585 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":354,"threads_started":1}
I20260929 15:44:14.129283 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.105561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44113 (local address 127.0.97.3:38325)
0929 15:44:14.106724 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.106784 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.109506 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.110585 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.110599 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.111282 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:14.112003 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.112027 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.113417 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.113424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.115276 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.115302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.116417 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.116424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.116689 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.117557 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:14.117590 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:14.120531 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:14.124625 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:14.124634 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:14.124653 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:14.125204 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:14.125680 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:14.125686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:14.125689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:14.125906 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:14.126481 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:14.126490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:14.126904 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.127213 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.127832 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":454,"threads_started":1}
I20260929 15:44:14.131664 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44113
I20260929 15:44:14.132037 1421 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:14.132912 1421 heartbeater.cc:511] Master 127.0.97.62:44113 requested a full tablet report, sending...
I20260929 15:44:14.134860 964 ts_manager.cc:194] Registered new tserver with Master: e92c9024b5974f4b8f890d4aae0abff0 (127.0.97.3:33529)
I20260929 15:44:14.135962 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:44:14.136072 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38325
I20260929 15:44:14.163142 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.144982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.97.62:44113)
0929 15:44:14.145173 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.145177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.145790 (+ 613us) server_negotiation.cc:407] Connection header received
0929 15:44:14.145903 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.145907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.146006 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.146172 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.148271 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.149054 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.151144 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.151687 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.151852 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.155679 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.155686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.155690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.155739 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.159373 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.160491 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.160495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.160498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.160649 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.161337 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.161342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.161345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.161809 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.162147 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.162597 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.162867 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:44:14.171941 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:14.183167 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:14.227969 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.201982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33529 (local address 127.0.0.1:36154)
0929 15:44:14.215042 (+ 13060us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.215069 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.215326 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.216893 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.216900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.216936 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.217492 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.217504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.220330 (+ 2826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.220336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.223095 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.223137 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.226347 (+ 3210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.226355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.226693 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.226830 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.226990 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.227315 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12935,"mutex_wait_us":117,"thread_start_us":10957,"threads_started":1}
I20260929 15:44:14.228229 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.201994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.97.1:46223)
0929 15:44:14.212721 (+ 10727us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.212731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.213847 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 15:44:14.214844 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.214856 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.215183 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.215615 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.217061 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.219458 (+ 2397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.223757 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.225293 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.225460 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.226265 (+ 805us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.227595 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.227810 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10532,"thread_start_us":10431,"threads_started":1}
I20260929 15:44:14.228655 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.202529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.0.97.3:33529)
0929 15:44:14.212361 (+ 9832us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.212376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.215211 (+ 2835us) server_negotiation.cc:407] Connection header received
0929 15:44:14.215739 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.215750 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.216125 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.216408 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.217988 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.220099 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.223452 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.225266 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.225460 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.225749 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.227215 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.228149 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9326,"thread_start_us":9093,"threads_started":1}
I20260929 15:44:14.228020 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.201804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46223 (local address 127.0.0.1:41612)
0929 15:44:14.213437 (+ 11633us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.213521 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.214034 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.216039 (+ 2005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.216048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.216090 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.216773 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.216791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.219724 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.219729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.223516 (+ 3787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.223533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.226408 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.226413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.226696 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.226817 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.226987 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.227383 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11199,"mutex_wait_us":831,"thread_start_us":12905,"threads_started":1}
I20260929 15:44:14.233163 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.215179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.97.2:35131)
0929 15:44:14.222831 (+ 7652us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.222842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.223759 (+ 917us) server_negotiation.cc:407] Connection header received
0929 15:44:14.223883 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.223892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.224277 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.224540 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.225243 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.226606 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.229496 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.231184 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.231620 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.231967 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.232546 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.232777 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7449,"thread_start_us":7323,"threads_started":1}
I20260929 15:44:14.233136 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.215411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35131 (local address 127.0.0.1:46436)
0929 15:44:14.223426 (+ 8015us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.223454 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.223620 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.224596 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.224603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.224635 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.225010 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.225019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.226813 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.226817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.229234 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.229251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.231728 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.231736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.232276 (+ 540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.232311 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.232475 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.232681 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7851,"thread_start_us":7669,"threads_started":1}
I20260929 15:44:14.234031 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 1e58e8aec4994f3fa7b95ac023d0ae30 (DEFAULT_TABLE table=test-table [id=e5ac56db02964c8f90764a6166ee970b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:14.234031 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 1e58e8aec4994f3fa7b95ac023d0ae30 (DEFAULT_TABLE table=test-table [id=e5ac56db02964c8f90764a6166ee970b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:14.235948 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e58e8aec4994f3fa7b95ac023d0ae30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:14.235948 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e58e8aec4994f3fa7b95ac023d0ae30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:14.238075 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 1e58e8aec4994f3fa7b95ac023d0ae30 (DEFAULT_TABLE table=test-table [id=e5ac56db02964c8f90764a6166ee970b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:14.240298 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e58e8aec4994f3fa7b95ac023d0ae30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:14.255254 1440 tablet_bootstrap.cc:492] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: Bootstrap starting.
I20260929 15:44:14.260108 1442 tablet_bootstrap.cc:492] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: Bootstrap starting.
I20260929 15:44:14.262424 1441 tablet_bootstrap.cc:492] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: Bootstrap starting.
I20260929 15:44:14.264287 1442 tablet_bootstrap.cc:654] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:14.265239 1440 tablet_bootstrap.cc:654] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:14.265756 1441 tablet_bootstrap.cc:654] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:14.265759 1442 log.cc:824] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:14.267094 1440 log.cc:824] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:14.267268 1441 log.cc:824] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:14.270835 1442 tablet_bootstrap.cc:492] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: No bootstrap required, opened a new log
I20260929 15:44:14.271404 1442 ts_tablet_manager.cc:1397] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 15:44:14.276291 1441 tablet_bootstrap.cc:492] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: No bootstrap required, opened a new log
I20260929 15:44:14.276291 1440 tablet_bootstrap.cc:492] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: No bootstrap required, opened a new log
I20260929 15:44:14.276721 1441 ts_tablet_manager.cc:1397] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260929 15:44:14.276721 1440 ts_tablet_manager.cc:1397] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.007s
I20260929 15:44:14.280952 1442 raft_consensus.cc:348] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.281486 1442 raft_consensus.cc:374] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:14.281585 1442 raft_consensus.cc:729] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abafdfa495c4115aa370710dbf7d225, State: Initialized, Role: FOLLOWER
I20260929 15:44:14.282368 1442 consensus_queue.cc:260] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.284803 1287 heartbeater.cc:503] Master 127.0.97.62:44113 was elected leader, sending a full tablet report...
I20260929 15:44:14.285684 1442 ts_tablet_manager.cc:1428] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 15:44:14.291442 1440 raft_consensus.cc:348] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.292389 1440 raft_consensus.cc:374] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:14.292541 1440 raft_consensus.cc:729] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea751778d284bca8441f115d9853c9f, State: Initialized, Role: FOLLOWER
I20260929 15:44:14.292219 1441 raft_consensus.cc:348] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.292743 1441 raft_consensus.cc:374] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:14.292861 1441 raft_consensus.cc:729] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c9024b5974f4b8f890d4aae0abff0, State: Initialized, Role: FOLLOWER
I20260929 15:44:14.293568 1440 consensus_queue.cc:260] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.293571 1441 consensus_queue.cc:260] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.295140 1421 heartbeater.cc:503] Master 127.0.97.62:44113 was elected leader, sending a full tablet report...
I20260929 15:44:14.296597 1441 ts_tablet_manager.cc:1428] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: Time spent starting tablet: real 0.020s user 0.007s sys 0.012s
I20260929 15:44:14.296595 1440 ts_tablet_manager.cc:1428] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260929 15:44:14.330729 1447 raft_consensus.cc:482] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:14.331240 1447 raft_consensus.cc:504] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.333922 1447 leader_election.cc:288] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92c9024b5974f4b8f890d4aae0abff0 (127.0.97.3:33529), 2abafdfa495c4115aa370710dbf7d225 (127.0.97.2:35131)
I20260929 15:44:14.338680 1446 raft_consensus.cc:482] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:14.339083 1446 raft_consensus.cc:504] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.346170 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.334438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33529 (local address 127.0.97.1:53351)
0929 15:44:14.335211 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.335237 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.335456 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.336082 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.336089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.336130 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.336815 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.336831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.338469 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.338474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.341740 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.341778 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.344813 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.344824 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.345108 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.345171 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.345298 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.345553 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":698,"threads_started":1}
I20260929 15:44:14.347537 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.335076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53351 (local address 127.0.97.3:33529)
0929 15:44:14.335393 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.335400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.335434 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:44:14.335650 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.335656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.335752 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.335905 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.337045 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.338250 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.342023 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.344129 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.344363 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.344551 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.346852 (+ 2301us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.347138 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:44:14.348717 1446 leader_election.cc:288] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ea751778d284bca8441f115d9853c9f (127.0.97.1:46223), e92c9024b5974f4b8f890d4aae0abff0 (127.0.97.3:33529)
I20260929 15:44:14.349260 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e58e8aec4994f3fa7b95ac023d0ae30" candidate_uuid: "2ea751778d284bca8441f115d9853c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c9024b5974f4b8f890d4aae0abff0" is_pre_election: true
I20260929 15:44:14.350126 1376 raft_consensus.cc:2454] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea751778d284bca8441f115d9853c9f in term 0.
I20260929 15:44:14.351435 1041 leader_election.cc:302] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea751778d284bca8441f115d9853c9f, e92c9024b5974f4b8f890d4aae0abff0; no voters:
I20260929 15:44:14.352419 1447 raft_consensus.cc:2781] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:14.352784 1447 raft_consensus.cc:482] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:14.352944 1447 raft_consensus.cc:3037] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:44:14.356853 1422 tablet.cc:2369] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:14.358697 1447 raft_consensus.cc:504] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.360322 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e58e8aec4994f3fa7b95ac023d0ae30" candidate_uuid: "2ea751778d284bca8441f115d9853c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c9024b5974f4b8f890d4aae0abff0"
I20260929 15:44:14.360647 1376 raft_consensus.cc:3037] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:14.362668 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.335714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35131 (local address 127.0.97.1:55263)
0929 15:44:14.337653 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.337675 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.337828 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.338241 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.338246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.338273 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.339914 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.339931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.341599 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.341605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.344553 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.344579 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.359576 (+ 14997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.359583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.359782 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.359827 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.359939 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.360143 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1770,"mutex_wait_us":4,"thread_start_us":1496,"threads_started":1}
I20260929 15:44:14.363845 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.335918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55263 (local address 127.0.97.2:35131)
0929 15:44:14.336248 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.336258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.337776 (+ 1518us) server_negotiation.cc:407] Connection header received
0929 15:44:14.338002 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.338007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.338081 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.338234 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.340159 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.341356 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.344873 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.347430 (+ 2557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.347685 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.348084 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.363261 (+ 15177us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.363467 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 15:44:14.364768 1376 raft_consensus.cc:2454] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea751778d284bca8441f115d9853c9f in term 1.
I20260929 15:44:14.365083 1447 leader_election.cc:288] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [CANDIDATE]: Term 1 election: Requested vote from peers e92c9024b5974f4b8f890d4aae0abff0 (127.0.97.3:33529), 2abafdfa495c4115aa370710dbf7d225 (127.0.97.2:35131)
I20260929 15:44:14.365523 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.343499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41639 (local address 127.0.97.3:33529)
0929 15:44:14.348877 (+ 5378us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.348887 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.357090 (+ 8203us) server_negotiation.cc:407] Connection header received
0929 15:44:14.357393 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.357399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.357552 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.357681 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.358506 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.359723 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.362665 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.364046 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.364208 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.364337 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.364992 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.365190 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5200,"thread_start_us":12423,"threads_started":1}
I20260929 15:44:14.365628 1041 leader_election.cc:302] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea751778d284bca8441f115d9853c9f, e92c9024b5974f4b8f890d4aae0abff0; no voters:
I20260929 15:44:14.366076 1447 raft_consensus.cc:2781] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:14.366459 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.342714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56823 (local address 127.0.97.1:46223)
0929 15:44:14.343014 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.343028 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.349805 (+ 6777us) server_negotiation.cc:407] Connection header received
0929 15:44:14.350225 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.350236 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.350366 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.350455 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:14.352051 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.359187 (+ 7136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.362477 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.363876 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.364256 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.364462 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.365887 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.366101 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:44:14.367834 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.342398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33529 (local address 127.0.97.2:41639)
0929 15:44:14.356860 (+ 14462us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.356895 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.357191 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.357722 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.357728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.357761 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.358252 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.358271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.359947 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.359952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.362465 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.362479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.364540 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.364548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.364768 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.364811 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.365821 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.366013 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14270,"mutex_wait_us":2,"thread_start_us":6578,"threads_started":1}
I20260929 15:44:14.367791 1447 raft_consensus.cc:686] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [term 1 LEADER]: Becoming Leader. State: Replica: 2ea751778d284bca8441f115d9853c9f, State: Running, Role: LEADER
I20260929 15:44:14.367724 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e58e8aec4994f3fa7b95ac023d0ae30" candidate_uuid: "2ea751778d284bca8441f115d9853c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2abafdfa495c4115aa370710dbf7d225" is_pre_election: true
I20260929 15:44:14.368201 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e58e8aec4994f3fa7b95ac023d0ae30" candidate_uuid: "2ea751778d284bca8441f115d9853c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2abafdfa495c4115aa370710dbf7d225"
I20260929 15:44:14.368484 1242 raft_consensus.cc:2454] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea751778d284bca8441f115d9853c9f in term 0.
I20260929 15:44:14.368800 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.342188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46223 (local address 127.0.97.2:56823)
0929 15:44:14.349628 (+ 7440us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:14.349660 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:14.350005 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:14.350760 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:14.350768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:14.350807 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:14.351706 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.351734 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.360622 (+ 8888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.360632 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.362272 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:14.362288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.365147 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.365155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:14.365424 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.365472 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 15:44:14.365652 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 15:44:14.365845 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7218,"mutex_wait_us":1,"thread_start_us":14228,"threads_started":1}
I20260929 15:44:14.368588 1447 consensus_queue.cc:237] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } }
I20260929 15:44:14.369832 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e58e8aec4994f3fa7b95ac023d0ae30" candidate_uuid: "2abafdfa495c4115aa370710dbf7d225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c9024b5974f4b8f890d4aae0abff0" is_pre_election: true
I20260929 15:44:14.370234 1376 raft_consensus.cc:2379] T 1e58e8aec4994f3fa7b95ac023d0ae30 P e92c9024b5974f4b8f890d4aae0abff0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2abafdfa495c4115aa370710dbf7d225 in current term 1: Already voted for candidate 2ea751778d284bca8441f115d9853c9f in this term.
I20260929 15:44:14.370302 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e58e8aec4994f3fa7b95ac023d0ae30" candidate_uuid: "2abafdfa495c4115aa370710dbf7d225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ea751778d284bca8441f115d9853c9f" is_pre_election: true
I20260929 15:44:14.371464 1178 leader_election.cc:302] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2abafdfa495c4115aa370710dbf7d225; no voters: 2ea751778d284bca8441f115d9853c9f, e92c9024b5974f4b8f890d4aae0abff0
I20260929 15:44:14.371991 1446 raft_consensus.cc:3037] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:14.375892 1446 raft_consensus.cc:2726] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 15:44:14.377139 963 catalog_manager.cc:5495] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ea751778d284bca8441f115d9853c9f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2ea751778d284bca8441f115d9853c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ea751778d284bca8441f115d9853c9f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92c9024b5974f4b8f890d4aae0abff0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2abafdfa495c4115aa370710dbf7d225" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35131 } health_report { overall_health: UNKNOWN } } }
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260929 15:44:14.410246 1154 tablet.cc:2369] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2ea751778d284bca8441f115d9853c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:14.416101 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.393732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.0.97.1:46223)
0929 15:44:14.394416 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.394425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.395499 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 15:44:14.395710 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.395715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.395818 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.395962 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.397941 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.398876 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.400500 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.400991 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.401121 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.406953 (+ 5832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.406971 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.406989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.407038 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.412282 (+ 5244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.413170 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.413178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.413181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.413548 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.414041 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.414046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.414048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.414497 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.414830 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.415195 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.415705 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696654, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696654, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:14.476457 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.458407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.97.1:46223)
0929 15:44:14.458708 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.458716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.459059 (+ 343us) server_negotiation.cc:407] Connection header received
0929 15:44:14.459537 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.459542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.459633 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.459758 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.461936 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.462752 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.464227 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.464758 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.464922 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.469572 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.469578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.469582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.469642 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.473169 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.474105 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.474109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.474112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.474288 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.474785 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.474789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.474791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.475152 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.475387 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.475817 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.476025 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260929 15:44:14.514096 1288 tablet.cc:2369] T 1e58e8aec4994f3fa7b95ac023d0ae30 P 2abafdfa495c4115aa370710dbf7d225: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:14.535056 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.514287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.0.97.1:46223)
0929 15:44:14.514597 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.514605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.515059 (+ 454us) server_negotiation.cc:407] Connection header received
0929 15:44:14.515324 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.515330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.515447 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.515602 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.518010 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.519379 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.520976 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.521651 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.521803 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.526362 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.526368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.526372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.526413 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.530381 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.531890 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.531897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.531899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.532084 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.532666 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.532670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.532672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.533111 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.533354 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.533881 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.534730 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260929 15:44:14.537475 1068 tablet_service.cc:2908] Rejecting scan request for tablet 1e58e8aec4994f3fa7b95ac023d0ae30: Uninitialized: clean time has not yet been initialized
I20260929 15:44:14.542172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1025
I20260929 15:44:14.559408 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.543566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.97.1:46223)
0929 15:44:14.543754 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.543759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.544135 (+ 376us) server_negotiation.cc:407] Connection header received
0929 15:44:14.544496 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.544502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.544662 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.544795 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.546107 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.546859 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.548242 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.548686 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.548808 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.552727 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.552734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.552738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.552790 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.556121 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.557008 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.557013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.557016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.557156 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.557889 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.557893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.557896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.558229 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.558491 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.558922 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.559147 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:44:14.560367 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:14.660995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1025
I20260929 15:44:14.681258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1159
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:14.698645 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.682259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.0.97.2:35131)
0929 15:44:14.682488 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.682496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.682518 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:44:14.682593 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.682597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.682671 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.682790 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.684022 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.684940 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.686341 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.686782 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.686946 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.691630 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.691650 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.691669 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.691720 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.695154 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.696059 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.696073 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.696076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.696386 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.696823 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.696830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.696833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.697266 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.697590 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.697933 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.698170 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:44:14.699661 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:14.811895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1159
I20260929 15:44:14.832274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1293
Sep 29 15:44:14 dist-test-slave-7491 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:14.852478 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:14.833144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.0.97.3:33529)
0929 15:44:14.833320 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:14.833324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:14.833344 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:44:14.833453 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:14.833457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:14.833547 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:14.833646 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:14.835247 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.836094 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.838411 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:14.839212 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:14.839349 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:14.844471 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:14.844502 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:14.844525 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:14.844583 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:14.848295 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.849254 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.849265 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.849267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.849476 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:14.850278 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:14.850282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:14.850285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:14.850789 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:14.851073 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:14.851595 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 15:44:14.851874 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:44:14.853312 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:14.954423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1293
I20260929 15:44:14.974403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 932
I20260929 15:44:14.976071 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:15.097237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 932
2026-09-29T15:44:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4791 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:15 dist-test-slave-7491 krb5kdc[1473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:15 dist-test-slave-7491 krb5kdc[1473](info): set up 2 sockets
Sep 29 15:44:15 dist-test-slave-7491 krb5kdc[1473](info): commencing operation
krb5kdc: starting...
W20260929 15:44:16.240960 388 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:16 dist-test-slave-7491 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:16.465266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33429
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:16.855259 1486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:16.855679 1486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:16.855784 1486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:16.864975 1486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:16.865126 1486 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:44:16.865170 1486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:16.865226 1486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:16.865268 1486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:16.865325 1486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:16.877293 1486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33429
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:16.879350 1486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:16.881594 1486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:16.897182 1492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:16.897339 1491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:16.897994 1494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:16.898454 1486 server_base.cc:1034] running on GCE node
I20260929 15:44:16.899572 1486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:16.901770 1486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:16.903187 1486 hybrid_clock.cc:648] HybridClock initialized: now 1790696656903086 us; error 57 us; skew 500 ppm
I20260929 15:44:16.903673 1486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:16 dist-test-slave-7491 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:16.907490 1486 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:16.909531 1486 webserver.cc:460] Webserver started at http://127.0.97.62:37187/ using document root <none> and password file <none>
I20260929 15:44:16.910503 1486 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:16.910616 1486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:16.911101 1486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:16.915339 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "337469e10ba949f4bf5f3d03bf9626fd"
format_stamp: "Formatted at 2026-09-29 15:44:16 on dist-test-slave-7491"
server_key: "9fd665f5287fc44b7fcb1e25a74681fe"
server_key_iv: "d4b4bd0647799132e779603b22103cb6"
server_key_version: "encryptionkey@0"
I20260929 15:44:16.916467 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "337469e10ba949f4bf5f3d03bf9626fd"
format_stamp: "Formatted at 2026-09-29 15:44:16 on dist-test-slave-7491"
server_key: "9fd665f5287fc44b7fcb1e25a74681fe"
server_key_iv: "d4b4bd0647799132e779603b22103cb6"
server_key_version: "encryptionkey@0"
I20260929 15:44:16.923440 1486 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:44:16.927827 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:16.929673 1486 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:16.929939 1486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "337469e10ba949f4bf5f3d03bf9626fd"
format_stamp: "Formatted at 2026-09-29 15:44:16 on dist-test-slave-7491"
server_key: "9fd665f5287fc44b7fcb1e25a74681fe"
server_key_iv: "d4b4bd0647799132e779603b22103cb6"
server_key_version: "encryptionkey@0"
I20260929 15:44:16.930346 1486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:16.956283 1486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:16.961819 1486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:16.962301 1486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:16.989091 1486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33429
I20260929 15:44:16.989121 1562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33429 every 8 connection(s)
I20260929 15:44:16.991313 1486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:16.992750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1486
I20260929 15:44:16.993085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:16.994156 388 external_mini_cluster.cc:1428] Setting key b5fc4fdf0255ee6155e1340f8d6cabd4
I20260929 15:44:16.997148 1563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:44:17 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:17.012631 1563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: Bootstrap starting.
I20260929 15:44:17.018522 1563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:17.020052 1563 log.cc:824] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:17.024231 1563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: No bootstrap required, opened a new log
I20260929 15:44:17.024796 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:16.996812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.0.97.62:33429)
0929 15:44:16.997868 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:16.997883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:16.997940 (+ 57us) server_negotiation.cc:407] Connection header received
0929 15:44:17.000349 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:17.000384 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:17.000912 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:17.001439 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:17.003144 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.004792 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.006156 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.007068 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.007287 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:17.012862 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:17.012877 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:17.012901 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:17.012959 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:17.018186 (+ 5227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:17.019415 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:17.019425 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:17.019435 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:17.019727 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:17.020482 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:17.020488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:17.020491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:17.021845 (+ 1354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:17.022222 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:17.022713 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 15:44:17.023237 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":526,"threads_started":1}
I20260929 15:44:17.033921 1563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337469e10ba949f4bf5f3d03bf9626fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260929 15:44:17.034372 1563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:17.034446 1563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337469e10ba949f4bf5f3d03bf9626fd, State: Initialized, Role: FOLLOWER
I20260929 15:44:17.035163 1563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337469e10ba949f4bf5f3d03bf9626fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260929 15:44:17.035457 1563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:17.035727 1563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:17.036132 1563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:17.074952 1563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337469e10ba949f4bf5f3d03bf9626fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260929 15:44:17.075829 1563 leader_election.cc:302] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 337469e10ba949f4bf5f3d03bf9626fd; no voters:
I20260929 15:44:17.076721 1563 leader_election.cc:288] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:17.076890 1568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:17.078576 1568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [term 1 LEADER]: Becoming Leader. State: Replica: 337469e10ba949f4bf5f3d03bf9626fd, State: Running, Role: LEADER
I20260929 15:44:17.079738 1563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:17.079644 1568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337469e10ba949f4bf5f3d03bf9626fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260929 15:44:17.084487 1570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 337469e10ba949f4bf5f3d03bf9626fd. Latest consensus state: current_term: 1 leader_uuid: "337469e10ba949f4bf5f3d03bf9626fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337469e10ba949f4bf5f3d03bf9626fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } } }
I20260929 15:44:17.084463 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "337469e10ba949f4bf5f3d03bf9626fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "337469e10ba949f4bf5f3d03bf9626fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } } }
I20260929 15:44:17.085256 1570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:17.085266 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:17.087524 1576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:17.093901 1576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:17.106693 1576 catalog_manager.cc:1347] Generated new cluster ID: f0217029498c46e1a956e7b8a920b6de
I20260929 15:44:17.106907 1576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:17.140138 1576 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:17.141834 1576 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:17.155845 1576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: Generated new TSK 0
I20260929 15:44:17.157366 1576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:17.256035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:17.634748 1589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:17.635131 1589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:17.635254 1589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:17.643388 1589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:17.643528 1589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:17.643620 1589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:44:17.643702 1589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:17.654402 1589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:17.655860 1589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:17.657483 1589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:17.673316 1594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:17.673264 1597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:17.673344 1595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:17.674711 1589 server_base.cc:1034] running on GCE node
I20260929 15:44:17.675784 1589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:17.677273 1589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:17.678785 1589 hybrid_clock.cc:648] HybridClock initialized: now 1790696657678662 us; error 88 us; skew 500 ppm
I20260929 15:44:17.679255 1589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:17 dist-test-slave-7491 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:17.683470 1589 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:17.685422 1589 webserver.cc:460] Webserver started at http://127.0.97.1:43755/ using document root <none> and password file <none>
I20260929 15:44:17.686278 1589 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:17.686442 1589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:17.686820 1589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:17.690296 1589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "2f32a0bd71ed4c6fbee9715607d1ff93"
format_stamp: "Formatted at 2026-09-29 15:44:17 on dist-test-slave-7491"
server_key: "98339f79d4547f0c6749e5edde66ebf3"
server_key_iv: "4718b4efae6ceb457e53b27523c2d7af"
server_key_version: "encryptionkey@0"
I20260929 15:44:17.691293 1589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2f32a0bd71ed4c6fbee9715607d1ff93"
format_stamp: "Formatted at 2026-09-29 15:44:17 on dist-test-slave-7491"
server_key: "98339f79d4547f0c6749e5edde66ebf3"
server_key_iv: "4718b4efae6ceb457e53b27523c2d7af"
server_key_version: "encryptionkey@0"
I20260929 15:44:17.697343 1589 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:44:17.702680 1604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:17.704433 1589 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:44:17.704659 1589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "2f32a0bd71ed4c6fbee9715607d1ff93"
format_stamp: "Formatted at 2026-09-29 15:44:17 on dist-test-slave-7491"
server_key: "98339f79d4547f0c6749e5edde66ebf3"
server_key_iv: "4718b4efae6ceb457e53b27523c2d7af"
server_key_version: "encryptionkey@0"
I20260929 15:44:17.704927 1589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:17.723829 1589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:17.730064 1589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:17.730479 1589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:17.734027 1589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:17.734166 1589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:17.734359 1589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:17.734458 1589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:44:17 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:17.765273 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:17.734737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54841 (local address 127.0.97.62:33429)
0929 15:44:17.737916 (+ 3179us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:17.737921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:17.737942 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:44:17.740779 (+ 2837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:17.740787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:17.740911 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:17.741073 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:17.743699 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.744870 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.748416 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.748996 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.749312 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:17.756749 (+ 7437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:17.756756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:17.756760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:17.756809 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:17.760404 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:17.761747 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:17.761752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:17.761755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:17.761921 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:17.762581 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:17.762586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:17.762590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:17.762924 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:17.763122 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:17.764476 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0929 15:44:17.764792 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3054,"thread_start_us":2948,"threads_started":1}
I20260929 15:44:17.767531 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:17.735707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.1:54841)
0929 15:44:17.736923 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:17.736992 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:17.740336 (+ 3344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:17.741419 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:17.741438 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:17.742380 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:17.743478 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:17.743502 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.745098 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.745109 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:17.748145 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:17.748198 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.749674 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.749704 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:17.750163 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:17.751309 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:17.751357 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:17.756115 (+ 4758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:17.760666 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:17.760685 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:17.760712 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:17.761533 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:17.762075 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:17.762081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:17.762085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:17.762402 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:17.763101 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:17.763112 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:17.763830 (+ 718us) client_negotiation.cc:769] Sending connection context
0929 15:44:17.764335 (+ 505us) client_negotiation.cc:240] Negotiation successful
0929 15:44:17.765292 (+ 957us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":486,"threads_started":1}
I20260929 15:44:17.805136 1589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42855
I20260929 15:44:17.805150 1724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42855 every 8 connection(s)
I20260929 15:44:17.806891 1589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:17.808830 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:17.787230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.1:47473)
0929 15:44:17.788188 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:17.788225 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:17.788529 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:17.789047 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:17.789052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:17.790137 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:17.790818 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:17.790841 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.792702 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.792708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:17.794576 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:17.794605 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.796232 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.796244 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:17.796569 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:17.797832 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:17.797856 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:17.799926 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:17.805132 (+ 5206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:17.805141 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:17.805146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:17.805881 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:17.806474 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:17.806479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:17.806483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:17.806655 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:17.807375 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:17.807379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:17.807757 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 15:44:17.807957 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 15:44:17.808423 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":501,"threads_started":1}
I20260929 15:44:17.809741 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:17.787262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47473 (local address 127.0.97.62:33429)
0929 15:44:17.787517 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:17.787527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:17.788381 (+ 854us) server_negotiation.cc:407] Connection header received
0929 15:44:17.788750 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:17.788757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:17.788862 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:17.789002 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:17.791150 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.792521 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.794889 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:17.795875 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:17.796088 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:17.800387 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:17.800394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:17.800424 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:17.800492 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:17.804866 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:17.806076 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:17.806082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:17.806086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:17.806268 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:17.806802 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:17.806810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:17.806816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:17.807218 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:17.807400 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:17.809129 (+ 1729us) server_negotiation.cc:299] Negotiation successful
0929 15:44:17.809378 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:44:17.809448 1725 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33429
I20260929 15:44:17.810365 1725 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:17.811785 1725 heartbeater.cc:511] Master 127.0.97.62:33429 requested a full tablet report, sending...
I20260929 15:44:17.811873 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1589
I20260929 15:44:17.812119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:17.813089 388 external_mini_cluster.cc:1428] Setting key b219b553fe7e55264d63cfc7f44cc1d9
I20260929 15:44:17.814931 1518 ts_manager.cc:194] Registered new tserver with Master: 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855)
I20260929 15:44:17.816947 1518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54841
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:17.863960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:18.273855 1729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:18.274320 1729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:18.274506 1729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:18.284298 1729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:18.284509 1729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:18.284703 1729 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:44:18.284866 1729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:18.298485 1729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:18.300010 1729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:18.301620 1729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:18.316891 1734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:18.317608 1735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:18.318054 1737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:18.318466 1729 server_base.cc:1034] running on GCE node
I20260929 15:44:18.319427 1729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:18.320837 1729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:18.322291 1729 hybrid_clock.cc:648] HybridClock initialized: now 1790696658322208 us; error 70 us; skew 500 ppm
I20260929 15:44:18.322729 1729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:18 dist-test-slave-7491 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:18.327488 1729 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:18.329493 1729 webserver.cc:460] Webserver started at http://127.0.97.2:41075/ using document root <none> and password file <none>
I20260929 15:44:18.330646 1729 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:18.330891 1729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:18.331377 1729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:18.335819 1729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "d6b0d1426fed441b9ecb398a3a331813"
format_stamp: "Formatted at 2026-09-29 15:44:18 on dist-test-slave-7491"
server_key: "961081701c9b66e347dca18d23d3a622"
server_key_iv: "4d5ed58d89abcb0f6e71672cacdb4f98"
server_key_version: "encryptionkey@0"
I20260929 15:44:18.336939 1729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d6b0d1426fed441b9ecb398a3a331813"
format_stamp: "Formatted at 2026-09-29 15:44:18 on dist-test-slave-7491"
server_key: "961081701c9b66e347dca18d23d3a622"
server_key_iv: "4d5ed58d89abcb0f6e71672cacdb4f98"
server_key_version: "encryptionkey@0"
I20260929 15:44:18.343348 1729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:18.348997 1744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:18.350891 1729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 15:44:18.351168 1729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "d6b0d1426fed441b9ecb398a3a331813"
format_stamp: "Formatted at 2026-09-29 15:44:18 on dist-test-slave-7491"
server_key: "961081701c9b66e347dca18d23d3a622"
server_key_iv: "4d5ed58d89abcb0f6e71672cacdb4f98"
server_key_version: "encryptionkey@0"
I20260929 15:44:18.351468 1729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:18.369230 1729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:18.375728 1729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:18.376135 1729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:18.380105 1729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:18.380338 1729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:44:18.380616 1729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:18.380712 1729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:44:18 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:18.416114 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:18.381338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33603 (local address 127.0.97.62:33429)
0929 15:44:18.385044 (+ 3706us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:18.385049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:18.385076 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:44:18.388831 (+ 3755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:18.388841 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:18.389020 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:18.389192 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:18.392559 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.394085 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.397318 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.398099 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.398293 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:18.405871 (+ 7578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:18.405883 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:18.405892 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:18.405961 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:18.411491 (+ 5530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:18.413298 (+ 1807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:18.413304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:18.413307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:18.413476 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:18.413992 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:18.413997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:18.414000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:18.414439 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:18.414629 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:18.415442 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 15:44:18.415707 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3577,"thread_start_us":3429,"threads_started":1}
I20260929 15:44:18.417519 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:18.382290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.2:33603)
0929 15:44:18.383818 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:18.383915 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:18.388283 (+ 4368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:18.389720 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:18.389743 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:18.390844 (+ 1101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:18.392137 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:18.392231 (+ 94us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.394331 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.394336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:18.397049 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:18.397079 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.398463 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.398474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:18.399177 (+ 703us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:18.400990 (+ 1813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:18.401084 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:18.405427 (+ 4343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:18.411881 (+ 6454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:18.411898 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:18.411935 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:18.413054 (+ 1119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:18.413619 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:18.413625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:18.413629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:18.413857 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:18.414601 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:18.414611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:18.415031 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 15:44:18.415340 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 15:44:18.415935 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":616,"threads_started":1}
I20260929 15:44:18.442440 1729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36685
I20260929 15:44:18.442550 1864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36685 every 8 connection(s)
I20260929 15:44:18.444250 1729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:18.447000 1865 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33429
I20260929 15:44:18.447590 1865 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:18.448515 1865 heartbeater.cc:511] Master 127.0.97.62:33429 requested a full tablet report, sending...
I20260929 15:44:18.448861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1729
I20260929 15:44:18.449059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:18.449618 388 external_mini_cluster.cc:1428] Setting key bc3aab5a36b14cc96df68ba709f98c08
I20260929 15:44:18.450467 1517 ts_manager.cc:194] Registered new tserver with Master: d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2:36685)
I20260929 15:44:18.451841 1517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33603
I20260929 15:44:18.457314 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:18.438534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36765 (local address 127.0.97.62:33429)
0929 15:44:18.438812 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:18.438822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:18.441007 (+ 2185us) server_negotiation.cc:407] Connection header received
0929 15:44:18.441280 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:18.441290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:18.441403 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:18.441523 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:18.443172 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.444344 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.446315 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.447026 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.447169 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:18.449691 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:18.449703 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:18.449709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:18.449773 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:18.453071 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:18.454356 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:18.454364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:18.454368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:18.454647 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:18.455191 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:18.455196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:18.455200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:18.455747 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:18.455919 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:18.456546 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 15:44:18.456855 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:44:18.457583 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:18.438696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.2:36765)
0929 15:44:18.440796 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:18.440830 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:18.441076 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:18.441725 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:18.441732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:18.442607 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:18.442986 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:18.442998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.444536 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.444543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:18.446115 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:18.446129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:18.447237 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:18.447246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:18.447476 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:18.448286 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:18.448306 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:18.449438 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:18.453447 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:18.453456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:18.453463 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:18.454132 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:18.454832 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:18.454840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:18.454846 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:18.455033 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:18.455894 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:18.455902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:18.456199 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 15:44:18.456397 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 15:44:18.457004 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1899,"thread_start_us":1687,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:18.501590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33429
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:44:18.820941 1725 heartbeater.cc:503] Master 127.0.97.62:33429 was elected leader, sending a full tablet report...
W20260929 15:44:18.906220 1869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:18.906591 1869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:18.906781 1869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:18.916034 1869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:18.916348 1869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:18.916471 1869 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:44:18.916560 1869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:18.929121 1869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:18.930560 1869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:18.932013 1869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:18.948012 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:18.948362 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:18.948534 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:18.949398 1869 server_base.cc:1034] running on GCE node
I20260929 15:44:18.950142 1869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:18.951246 1869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:18.952534 1869 hybrid_clock.cc:648] HybridClock initialized: now 1790696658952478 us; error 42 us; skew 500 ppm
I20260929 15:44:18.952909 1869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:18 dist-test-slave-7491 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:18.957216 1869 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:18.959213 1869 webserver.cc:460] Webserver started at http://127.0.97.3:42993/ using document root <none> and password file <none>
I20260929 15:44:18.960021 1869 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:18.960121 1869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:18.960503 1869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:18.963778 1869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "e6dffe1170614540a8b42bb1b886ddaf"
format_stamp: "Formatted at 2026-09-29 15:44:18 on dist-test-slave-7491"
server_key: "e5ea3804fec2d448503842e76a349752"
server_key_iv: "29a047afde9711da669b9f11e064db6f"
server_key_version: "encryptionkey@0"
I20260929 15:44:18.964704 1869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e6dffe1170614540a8b42bb1b886ddaf"
format_stamp: "Formatted at 2026-09-29 15:44:18 on dist-test-slave-7491"
server_key: "e5ea3804fec2d448503842e76a349752"
server_key_iv: "29a047afde9711da669b9f11e064db6f"
server_key_version: "encryptionkey@0"
I20260929 15:44:18.970841 1869 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:18.976276 1884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:18.978297 1869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 15:44:18.978571 1869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "e6dffe1170614540a8b42bb1b886ddaf"
format_stamp: "Formatted at 2026-09-29 15:44:18 on dist-test-slave-7491"
server_key: "e5ea3804fec2d448503842e76a349752"
server_key_iv: "29a047afde9711da669b9f11e064db6f"
server_key_version: "encryptionkey@0"
I20260929 15:44:18.979017 1869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:19.007018 1869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:19.013653 1869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:19.013989 1869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:19.017352 1869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:19.017483 1869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:19.017649 1869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:19.017730 1869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:44:19 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:19.048786 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.019518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.3:40647)
0929 15:44:19.020782 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.020833 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.023461 (+ 2628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.024632 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.024653 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.025839 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:19.027251 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.027290 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.028844 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.028852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.030988 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.031010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.032097 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.032104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.032615 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.033654 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:19.033680 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:19.037791 (+ 4111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:19.043021 (+ 5230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:19.043034 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:19.043058 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:19.043801 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:19.044292 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:19.044297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:19.044299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:19.044524 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:19.045435 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:19.045443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:19.046032 (+ 589us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.046388 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.047083 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":705,"threads_started":1}
I20260929 15:44:19.049331 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.018539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40647 (local address 127.0.97.62:33429)
0929 15:44:19.019196 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.019204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.022696 (+ 3492us) server_negotiation.cc:407] Connection header received
0929 15:44:19.023957 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.023963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.024082 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.024276 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:19.027613 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.028672 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.031178 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.031805 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.031957 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.038105 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:19.038112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:19.038116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:19.038154 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:19.042716 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.043951 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.043959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.043962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.044140 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.044685 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.044690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.044693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.045157 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:19.045349 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.048463 (+ 3114us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.048837 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":317,"threads_started":1}
I20260929 15:44:19.064087 1869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34371
I20260929 15:44:19.064157 2004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34371 every 8 connection(s)
I20260929 15:44:19.065908 1869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:19.068645 2005 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33429
I20260929 15:44:19.068928 2005 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:19.069763 2005 heartbeater.cc:511] Master 127.0.97.62:33429 requested a full tablet report, sending...
I20260929 15:44:19.072033 1517 ts_manager.cc:194] Registered new tserver with Master: e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371)
I20260929 15:44:19.073534 1517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40647
I20260929 15:44:19.076409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 1869
I20260929 15:44:19.076656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:19.077039 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.057368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59127 (local address 127.0.97.62:33429)
0929 15:44:19.057632 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.057643 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.058533 (+ 890us) server_negotiation.cc:407] Connection header received
0929 15:44:19.059023 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.059031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.059146 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.059297 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:19.061009 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.062205 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.064689 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.065319 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.065477 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.068146 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:19.068156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:19.068203 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:19.068264 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:19.073204 (+ 4940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.074380 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.074386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.074391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.074595 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.075233 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.075237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.075240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.075591 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:19.075743 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.076331 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.076647 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:44:19.077044 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.057368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.97.3:59127)
0929 15:44:19.058125 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.058335 (+ 210us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.058778 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.059385 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.059395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.060248 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:19.060775 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.060791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.062438 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.062442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.064519 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.064536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.065614 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.065623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.065821 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.066609 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:19.066642 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:19.067803 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:19.073474 (+ 5671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:19.073478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:19.073482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:19.074093 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:19.074804 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:19.074809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:19.074813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:19.075018 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:19.075756 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:19.075761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:19.075965 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.076143 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.076550 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":439,"threads_started":1}
I20260929 15:44:19.077335 388 external_mini_cluster.cc:1428] Setting key cfc0122ed4e8fe627a1268cd401ebd78
I20260929 15:44:19.081359 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:44:19 dist-test-slave-7491 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:44:19 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:19.137212 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.118489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.0.97.62:33429)
0929 15:44:19.118709 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.118715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.119073 (+ 358us) server_negotiation.cc:407] Connection header received
0929 15:44:19.119200 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.119204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.119271 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.119373 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:19.121175 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.121952 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.123942 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.124482 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.124603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.129674 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:19.129680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:19.129684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:19.129729 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:19.133543 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.134510 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.134515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.134518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.134784 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.135372 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.135378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.135381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.135761 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:19.136055 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.136653 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.136897 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:44:19.143277 1518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:19.150836 1518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:19.184593 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.169227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36685 (local address 127.0.0.1:41898)
0929 15:44:19.170338 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.170397 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.170663 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.171982 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.171994 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.172037 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.172822 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.172840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.175970 (+ 3130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.175982 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.179510 (+ 3528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.179555 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.183008 (+ 3453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.183018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.183408 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.183474 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.183627 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.183845 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":470,"threads_started":1}
I20260929 15:44:19.184650 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.169502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.0.97.2:36685)
0929 15:44:19.170312 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.170323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.170910 (+ 587us) server_negotiation.cc:407] Connection header received
0929 15:44:19.171049 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.171057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.171364 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.171631 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.173085 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.175557 (+ 2472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.179932 (+ 4375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.182078 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.182306 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.182760 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.184042 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.184298 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":1150,"threads_started":1}
I20260929 15:44:19.189580 1795 tablet_service.cc:1467] Processing CreateTablet for tablet b03b0e6e80724061991a1bf2080adfda (DEFAULT_TABLE table=test-table [id=a7e4eb2deac641c59820abc59926c547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:19.191174 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03b0e6e80724061991a1bf2080adfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.192092 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.170073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34371 (local address 127.0.0.1:39222)
0929 15:44:19.171006 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.171025 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.171199 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.184258 (+ 13059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.184263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.184288 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.185092 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.185105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.186826 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.186831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.189131 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.189146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.191149 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.191156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.191353 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.191390 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.191480 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.191632 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":26,"thread_start_us":2452,"threads_started":1}
I20260929 15:44:19.192282 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.171274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42855 (local address 127.0.0.1:34928)
0929 15:44:19.182563 (+ 11289us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.182583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.182828 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.183959 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.183963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.183997 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.184870 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.184884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.186761 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.186769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.189304 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.189319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.191479 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.191486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.191632 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.191659 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.191743 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.191888 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11150,"mutex_wait_us":62,"thread_start_us":10993,"threads_started":1}
I20260929 15:44:19.193017 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.171871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.97.1:42855)
0929 15:44:19.181731 (+ 9860us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.181746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.182696 (+ 950us) server_negotiation.cc:407] Connection header received
0929 15:44:19.183037 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.183047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.183518 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.183733 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.185076 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.186552 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.189488 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.190962 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.191101 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.191329 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.192526 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.192738 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9574,"thread_start_us":9885,"threads_started":1}
I20260929 15:44:19.193302 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.170410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.97.3:34371)
0929 15:44:19.183400 (+ 12990us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.183408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.183443 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:44:19.183541 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.183547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.184003 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.184438 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.185347 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.186655 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.189311 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.190894 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.192487 (+ 1593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.192726 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.192793 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.192959 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12803,"thread_start_us":13338,"threads_started":1}
I20260929 15:44:19.197676 1655 tablet_service.cc:1467] Processing CreateTablet for tablet b03b0e6e80724061991a1bf2080adfda (DEFAULT_TABLE table=test-table [id=a7e4eb2deac641c59820abc59926c547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:19.197897 1935 tablet_service.cc:1467] Processing CreateTablet for tablet b03b0e6e80724061991a1bf2080adfda (DEFAULT_TABLE table=test-table [id=a7e4eb2deac641c59820abc59926c547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:19.199105 1655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03b0e6e80724061991a1bf2080adfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.199627 1935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03b0e6e80724061991a1bf2080adfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.214504 2025 tablet_bootstrap.cc:492] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Bootstrap starting.
I20260929 15:44:19.217149 2024 tablet_bootstrap.cc:492] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Bootstrap starting.
I20260929 15:44:19.218668 2026 tablet_bootstrap.cc:492] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Bootstrap starting.
I20260929 15:44:19.219247 2025 tablet_bootstrap.cc:654] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.221040 2025 log.cc:824] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:19.221545 2024 tablet_bootstrap.cc:654] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.222909 2024 log.cc:824] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:19.223085 2026 tablet_bootstrap.cc:654] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.224933 2026 log.cc:824] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:19.226173 2025 tablet_bootstrap.cc:492] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: No bootstrap required, opened a new log
I20260929 15:44:19.226320 2024 tablet_bootstrap.cc:492] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: No bootstrap required, opened a new log
I20260929 15:44:19.226614 2024 ts_tablet_manager.cc:1397] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 15:44:19.226665 2025 ts_tablet_manager.cc:1397] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 15:44:19.229532 2026 tablet_bootstrap.cc:492] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: No bootstrap required, opened a new log
I20260929 15:44:19.230019 2026 ts_tablet_manager.cc:1397] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 15:44:19.235358 2024 raft_consensus.cc:348] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.235824 2024 raft_consensus.cc:374] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.235925 2024 raft_consensus.cc:729] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dffe1170614540a8b42bb1b886ddaf, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.236578 2024 consensus_queue.cc:260] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.237826 2005 heartbeater.cc:503] Master 127.0.97.62:33429 was elected leader, sending a full tablet report...
I20260929 15:44:19.238617 2024 ts_tablet_manager.cc:1428] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260929 15:44:19.240788 2025 raft_consensus.cc:348] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.241323 2025 raft_consensus.cc:374] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.241427 2025 raft_consensus.cc:729] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32a0bd71ed4c6fbee9715607d1ff93, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.242136 2025 consensus_queue.cc:260] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.243629 2026 raft_consensus.cc:348] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.244128 2026 raft_consensus.cc:374] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.244264 2025 ts_tablet_manager.cc:1428] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Time spent starting tablet: real 0.017s user 0.016s sys 0.004s
I20260929 15:44:19.244263 2026 raft_consensus.cc:729] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b0d1426fed441b9ecb398a3a331813, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.245011 2026 consensus_queue.cc:260] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.247619 1865 heartbeater.cc:503] Master 127.0.97.62:33429 was elected leader, sending a full tablet report...
I20260929 15:44:19.248729 2026 ts_tablet_manager.cc:1428] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Time spent starting tablet: real 0.019s user 0.015s sys 0.005s
W20260929 15:44:19.310292 1726 tablet.cc:2369] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:44:19.321727 2006 tablet.cc:2369] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:44:19.448308 1866 tablet.cc:2369] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:19.498869 2030 raft_consensus.cc:482] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:19.499423 2030 raft_consensus.cc:504] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.501696 2030 leader_election.cc:288] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2:36685), 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855)
I20260929 15:44:19.511536 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.502898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55933 (local address 127.0.97.1:42855)
0929 15:44:19.503098 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.503102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.503752 (+ 650us) server_negotiation.cc:407] Connection header received
0929 15:44:19.503910 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.503917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.504002 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.504107 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.504766 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.505531 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.508430 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.510030 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.510258 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.510398 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.510870 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.511056 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:44:19.511548 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.502162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36685 (local address 127.0.97.3:35589)
0929 15:44:19.502375 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.502402 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.502727 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.503229 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.503237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.503284 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.503542 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.503554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.505111 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.505118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.507642 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.507661 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.510531 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.510536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.510735 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.510797 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.510922 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.511079 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":35}
I20260929 15:44:19.511610 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.502273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35589 (local address 127.0.97.2:36685)
0929 15:44:19.502488 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.502492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.502587 (+ 95us) server_negotiation.cc:407] Connection header received
0929 15:44:19.502968 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.502974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.503054 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.503165 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.503805 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.504880 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.508426 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.510030 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.510274 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.510398 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.511015 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.511333 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:44:19.511653 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.502847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42855 (local address 127.0.97.3:55933)
0929 15:44:19.503615 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.503637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.503780 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.504241 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.504247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.504280 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.504616 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.504625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.505725 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.505734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.507963 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.507973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.510465 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.510471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.510640 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.510673 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.510801 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.510982 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":44,"thread_start_us":393,"threads_started":1}
I20260929 15:44:19.513243 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03b0e6e80724061991a1bf2080adfda" candidate_uuid: "e6dffe1170614540a8b42bb1b886ddaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b0d1426fed441b9ecb398a3a331813" is_pre_election: true
I20260929 15:44:19.513243 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03b0e6e80724061991a1bf2080adfda" candidate_uuid: "e6dffe1170614540a8b42bb1b886ddaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" is_pre_election: true
I20260929 15:44:19.513926 1675 raft_consensus.cc:2454] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dffe1170614540a8b42bb1b886ddaf in term 0.
I20260929 15:44:19.514021 1815 raft_consensus.cc:2454] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dffe1170614540a8b42bb1b886ddaf in term 0.
I20260929 15:44:19.514978 1887 leader_election.cc:302] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a0bd71ed4c6fbee9715607d1ff93, e6dffe1170614540a8b42bb1b886ddaf; no voters:
I20260929 15:44:19.515616 2030 raft_consensus.cc:2781] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:19.515839 2030 raft_consensus.cc:482] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:19.515988 2030 raft_consensus.cc:3037] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:19.519304 2030 raft_consensus.cc:504] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.520406 2030 leader_election.cc:288] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [CANDIDATE]: Term 1 election: Requested vote from peers d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2:36685), 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855)
I20260929 15:44:19.521301 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03b0e6e80724061991a1bf2080adfda" candidate_uuid: "e6dffe1170614540a8b42bb1b886ddaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b0d1426fed441b9ecb398a3a331813"
I20260929 15:44:19.521399 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03b0e6e80724061991a1bf2080adfda" candidate_uuid: "e6dffe1170614540a8b42bb1b886ddaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93"
I20260929 15:44:19.521555 1815 raft_consensus.cc:3037] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:19.521680 1675 raft_consensus.cc:3037] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:19.524021 1675 raft_consensus.cc:2454] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dffe1170614540a8b42bb1b886ddaf in term 1.
I20260929 15:44:19.524272 1815 raft_consensus.cc:2454] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dffe1170614540a8b42bb1b886ddaf in term 1.
I20260929 15:44:19.524649 1887 leader_election.cc:302] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a0bd71ed4c6fbee9715607d1ff93, e6dffe1170614540a8b42bb1b886ddaf; no voters:
I20260929 15:44:19.524971 2030 raft_consensus.cc:2781] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:19.525486 2030 raft_consensus.cc:686] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 1 LEADER]: Becoming Leader. State: Replica: e6dffe1170614540a8b42bb1b886ddaf, State: Running, Role: LEADER
I20260929 15:44:19.526283 2030 consensus_queue.cc:237] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.532625 1516 catalog_manager.cc:5495] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e6dffe1170614540a8b42bb1b886ddaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } health_report { overall_health: HEALTHY } } }
I20260929 15:44:19.576813 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.570534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.97.3:34371)
0929 15:44:19.570815 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.570821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.570845 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:44:19.571211 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.571215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.571308 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.571462 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:19.571855 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.572693 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.574189 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.574628 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.574792 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.575398 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:19.576025 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:19.576268 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.576416 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.576509 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:44:19.586424 1675 raft_consensus.cc:1261] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Refusing update from remote peer e6dffe1170614540a8b42bb1b886ddaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:19.586445 1815 raft_consensus.cc:1261] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Refusing update from remote peer e6dffe1170614540a8b42bb1b886ddaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:19.587467 2030 consensus_queue.cc:1035] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:19.587878 2034 consensus_queue.cc:1035] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:19.633046 1517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:19.633328 1517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:19.639340 1517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: Sending DeleteTablet for 3 replicas of tablet b03b0e6e80724061991a1bf2080adfda
I20260929 15:44:19.641377 1795 tablet_service.cc:1514] Processing DeleteTablet for tablet b03b0e6e80724061991a1bf2080adfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41898
I20260929 15:44:19.641377 1655 tablet_service.cc:1514] Processing DeleteTablet for tablet b03b0e6e80724061991a1bf2080adfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34928
I20260929 15:44:19.641494 1935 tablet_service.cc:1514] Processing DeleteTablet for tablet b03b0e6e80724061991a1bf2080adfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39222
I20260929 15:44:19.646245 1517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:19.647997 1517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:19.656910 2048 tablet_replica.cc:330] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: stopping tablet replica
I20260929 15:44:19.657634 2048 raft_consensus.cc:2229] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:19.658782 2048 raft_consensus.cc:2258] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:19.659767 2049 tablet_replica.cc:330] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: stopping tablet replica
I20260929 15:44:19.660363 2049 raft_consensus.cc:2229] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:19.660960 2049 raft_consensus.cc:2258] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:19.662968 1934 tablet_service.cc:1467] Processing CreateTablet for tablet 8a04455f85024342a88672573cef6ff5 (DEFAULT_TABLE table=test-table [id=68116134aa9a4187963c7c0960bf8dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:19.663508 2049 ts_tablet_manager.cc:1905] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:19.663815 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a04455f85024342a88672573cef6ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.665055 2047 tablet_replica.cc:330] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: stopping tablet replica
I20260929 15:44:19.665529 2047 raft_consensus.cc:2229] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:19.665940 1795 tablet_service.cc:1467] Processing CreateTablet for tablet 8a04455f85024342a88672573cef6ff5 (DEFAULT_TABLE table=test-table [id=68116134aa9a4187963c7c0960bf8dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:19.666319 2047 raft_consensus.cc:2258] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:19.665596 1655 tablet_service.cc:1467] Processing CreateTablet for tablet 8a04455f85024342a88672573cef6ff5 (DEFAULT_TABLE table=test-table [id=68116134aa9a4187963c7c0960bf8dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:19.668310 2048 ts_tablet_manager.cc:1905] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:19.668339 1655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a04455f85024342a88672573cef6ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.668529 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a04455f85024342a88672573cef6ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.672315 2024 tablet_bootstrap.cc:492] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Bootstrap starting.
I20260929 15:44:19.674304 2024 tablet_bootstrap.cc:654] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.681007 2026 tablet_bootstrap.cc:492] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Bootstrap starting.
I20260929 15:44:19.683637 2026 tablet_bootstrap.cc:654] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.684077 2024 tablet_bootstrap.cc:492] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: No bootstrap required, opened a new log
I20260929 15:44:19.684373 2024 ts_tablet_manager.cc:1397] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 15:44:19.684487 2049 ts_tablet_manager.cc:1918] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:19.684665 2049 log.cc:1196] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/b03b0e6e80724061991a1bf2080adfda
I20260929 15:44:19.685475 2049 ts_tablet_manager.cc:1939] T b03b0e6e80724061991a1bf2080adfda P d6b0d1426fed441b9ecb398a3a331813: Deleting consensus metadata
I20260929 15:44:19.685321 2024 raft_consensus.cc:348] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.685585 2024 raft_consensus.cc:374] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.685678 2024 raft_consensus.cc:729] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dffe1170614540a8b42bb1b886ddaf, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.685986 2024 consensus_queue.cc:260] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.686434 2047 ts_tablet_manager.cc:1905] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:19.686563 2024 ts_tablet_manager.cc:1428] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:44:19.687034 2048 ts_tablet_manager.cc:1918] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:19.687219 2048 log.cc:1196] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/b03b0e6e80724061991a1bf2080adfda
I20260929 15:44:19.687808 2048 ts_tablet_manager.cc:1939] T b03b0e6e80724061991a1bf2080adfda P 2f32a0bd71ed4c6fbee9715607d1ff93: Deleting consensus metadata
I20260929 15:44:19.687966 1504 catalog_manager.cc:4841] TS d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2:36685): tablet b03b0e6e80724061991a1bf2080adfda (table test-table [id=a7e4eb2deac641c59820abc59926c547]) successfully deleted
I20260929 15:44:19.689917 1504 catalog_manager.cc:4841] TS 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855): tablet b03b0e6e80724061991a1bf2080adfda (table test-table [id=a7e4eb2deac641c59820abc59926c547]) successfully deleted
I20260929 15:44:19.691999 2025 tablet_bootstrap.cc:492] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Bootstrap starting.
I20260929 15:44:19.694363 2025 tablet_bootstrap.cc:654] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.695941 2047 ts_tablet_manager.cc:1918] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:19.696267 2047 log.cc:1196] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/b03b0e6e80724061991a1bf2080adfda
I20260929 15:44:19.697444 2047 ts_tablet_manager.cc:1939] T b03b0e6e80724061991a1bf2080adfda P e6dffe1170614540a8b42bb1b886ddaf: Deleting consensus metadata
I20260929 15:44:19.700098 1503 catalog_manager.cc:4841] TS e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371): tablet b03b0e6e80724061991a1bf2080adfda (table test-table [id=a7e4eb2deac641c59820abc59926c547]) successfully deleted
I20260929 15:44:19.702380 2026 tablet_bootstrap.cc:492] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: No bootstrap required, opened a new log
I20260929 15:44:19.702564 2026 ts_tablet_manager.cc:1397] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20260929 15:44:19.703460 2026 raft_consensus.cc:348] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.703727 2026 raft_consensus.cc:374] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.703826 2026 raft_consensus.cc:729] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b0d1426fed441b9ecb398a3a331813, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.704173 2026 consensus_queue.cc:260] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.704746 2025 tablet_bootstrap.cc:492] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: No bootstrap required, opened a new log
I20260929 15:44:19.704914 2025 ts_tablet_manager.cc:1397] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260929 15:44:19.705487 2026 ts_tablet_manager.cc:1428] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:44:19.705825 2025 raft_consensus.cc:348] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.706058 2025 raft_consensus.cc:374] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.706161 2025 raft_consensus.cc:729] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32a0bd71ed4c6fbee9715607d1ff93, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.706457 2025 consensus_queue.cc:260] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.707183 2025 ts_tablet_manager.cc:1428] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:44:19.729669 2032 raft_consensus.cc:482] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:19.729982 2032 raft_consensus.cc:504] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.732048 2032 leader_election.cc:288] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855), e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371)
I20260929 15:44:19.740818 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.732379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42855 (local address 127.0.97.2:55421)
0929 15:44:19.733225 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.733246 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.733446 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.733988 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.733994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.734024 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.734506 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.734521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.736454 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.736461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.738260 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.738275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.739686 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.739695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.739946 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.739986 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.740106 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.740332 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":4,"thread_start_us":506,"threads_started":1}
I20260929 15:44:19.740866 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.732444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55421 (local address 127.0.97.1:42855)
0929 15:44:19.732638 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.732645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.733333 (+ 688us) server_negotiation.cc:407] Connection header received
0929 15:44:19.733612 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.733618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.733708 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.733773 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.734771 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.736248 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.738471 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.739359 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.739539 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.739714 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.740229 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.740463 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:44:19.742841 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.733297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52771 (local address 127.0.97.3:34371)
0929 15:44:19.733541 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.733549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.734133 (+ 584us) server_negotiation.cc:407] Connection header received
0929 15:44:19.734696 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.734710 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.734805 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.734907 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:19.735627 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.736935 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.739997 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.741419 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.741549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.741667 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.742311 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.742488 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:44:19.742727 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a04455f85024342a88672573cef6ff5" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" is_pre_election: true
I20260929 15:44:19.743171 1674 raft_consensus.cc:2454] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 0.
I20260929 15:44:19.743407 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.732665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34371 (local address 127.0.97.2:52771)
0929 15:44:19.734024 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.734038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.734414 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.734977 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.734982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.735019 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:19.735385 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.735396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.737144 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.737151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.739708 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.739735 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.741788 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.741794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.741996 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.742033 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.742157 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.742763 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"thread_start_us":1076,"threads_started":1}
I20260929 15:44:19.743963 1747 leader_election.cc:302] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a0bd71ed4c6fbee9715607d1ff93, d6b0d1426fed441b9ecb398a3a331813; no voters:
I20260929 15:44:19.744648 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a04455f85024342a88672573cef6ff5" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dffe1170614540a8b42bb1b886ddaf" is_pre_election: true
I20260929 15:44:19.745326 1955 raft_consensus.cc:2454] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 0.
I20260929 15:44:19.746030 2032 raft_consensus.cc:2781] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:19.746200 2032 raft_consensus.cc:482] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:19.746322 2032 raft_consensus.cc:3037] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:19.749127 2032 raft_consensus.cc:504] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.750366 2032 leader_election.cc:288] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 election: Requested vote from peers 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855), e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371)
I20260929 15:44:19.751320 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a04455f85024342a88672573cef6ff5" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93"
I20260929 15:44:19.751451 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a04455f85024342a88672573cef6ff5" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dffe1170614540a8b42bb1b886ddaf"
I20260929 15:44:19.751616 1674 raft_consensus.cc:3037] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:19.751757 1955 raft_consensus.cc:3037] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:19.755492 1674 raft_consensus.cc:2454] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 1.
I20260929 15:44:19.756244 1747 leader_election.cc:302] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a0bd71ed4c6fbee9715607d1ff93, d6b0d1426fed441b9ecb398a3a331813; no voters:
I20260929 15:44:19.756575 1955 raft_consensus.cc:2454] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 1.
I20260929 15:44:19.756681 2032 raft_consensus.cc:2781] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:19.757671 2032 raft_consensus.cc:686] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 1 LEADER]: Becoming Leader. State: Replica: d6b0d1426fed441b9ecb398a3a331813, State: Running, Role: LEADER
I20260929 15:44:19.758239 2032 consensus_queue.cc:237] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.763384 1516 catalog_manager.cc:5495] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } health_report { overall_health: UNKNOWN } } }
Sep 29 15:44:19 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:19.834043 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.811767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.0.97.62:33429)
0929 15:44:19.812918 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:19.812925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:19.814161 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 15:44:19.814434 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:19.814441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:19.814577 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:19.814736 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:19.816478 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.818075 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.820475 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.821195 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.821451 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.826686 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:19.826694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:19.826699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:19.826740 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:19.830735 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.831832 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.831845 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.831848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.831982 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:19.832427 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:19.832431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:19.832433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:19.832723 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:19.832858 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:19.833369 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 15:44:19.833734 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":970,"thread_start_us":765,"threads_started":1}
I20260929 15:44:19.834090 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:19.812810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33429 (local address 127.0.0.1:34460)
0929 15:44:19.813971 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:19.813995 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:19.814252 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:19.814863 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:19.814868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:19.815836 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:19.816122 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.816135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.818295 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.818302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.820161 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:19.820230 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:19.821401 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:19.821408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:19.821641 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:19.823038 (+ 1397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:19.823063 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:19.826388 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:19.830937 (+ 4549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:19.830956 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:19.830966 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:19.831659 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:19.832099 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:19.832103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:19.832107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:19.832307 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:19.832863 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:19.832873 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:19.833081 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 15:44:19.833275 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 15:44:19.833621 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":29,"thread_start_us":863,"threads_started":1}
I20260929 15:44:19.840730 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34460:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:44:19.841862 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:19.849280 1795 tablet_service.cc:1467] Processing CreateTablet for tablet 75e3f1dc729c4ec79ebdbce8399a1878 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03c624f82c704972921dac3d8041cf00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:44:19.849212 1655 tablet_service.cc:1467] Processing CreateTablet for tablet 75e3f1dc729c4ec79ebdbce8399a1878 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03c624f82c704972921dac3d8041cf00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:44:19.849843 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e3f1dc729c4ec79ebdbce8399a1878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.849854 1655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e3f1dc729c4ec79ebdbce8399a1878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.853314 1934 tablet_service.cc:1467] Processing CreateTablet for tablet 75e3f1dc729c4ec79ebdbce8399a1878 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03c624f82c704972921dac3d8041cf00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:44:19.853940 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e3f1dc729c4ec79ebdbce8399a1878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:19.858254 2026 tablet_bootstrap.cc:492] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: Bootstrap starting.
I20260929 15:44:19.860719 2026 tablet_bootstrap.cc:654] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.862285 2025 tablet_bootstrap.cc:492] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: Bootstrap starting.
I20260929 15:44:19.864499 2025 tablet_bootstrap.cc:654] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.866850 2024 tablet_bootstrap.cc:492] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: Bootstrap starting.
I20260929 15:44:19.869467 2024 tablet_bootstrap.cc:654] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:19.875432 2026 tablet_bootstrap.cc:492] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: No bootstrap required, opened a new log
I20260929 15:44:19.875494 2024 tablet_bootstrap.cc:492] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: No bootstrap required, opened a new log
I20260929 15:44:19.875710 2026 ts_tablet_manager.cc:1397] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260929 15:44:19.877125 2025 tablet_bootstrap.cc:492] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: No bootstrap required, opened a new log
I20260929 15:44:19.876917 2026 raft_consensus.cc:348] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.877327 2025 ts_tablet_manager.cc:1397] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260929 15:44:19.877394 2026 raft_consensus.cc:374] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.877528 2026 raft_consensus.cc:729] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b0d1426fed441b9ecb398a3a331813, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.875986 2024 ts_tablet_manager.cc:1397] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260929 15:44:19.877892 2026 consensus_queue.cc:260] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.878440 2032 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:19.878818 2032 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:19.879403 2024 raft_consensus.cc:348] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.879786 2024 raft_consensus.cc:374] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.879938 2024 raft_consensus.cc:729] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dffe1170614540a8b42bb1b886ddaf, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.879904 2025 raft_consensus.cc:348] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.880564 2025 raft_consensus.cc:374] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:19.880683 2025 raft_consensus.cc:729] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32a0bd71ed4c6fbee9715607d1ff93, State: Initialized, Role: FOLLOWER
I20260929 15:44:19.880992 2025 consensus_queue.cc:260] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.881004 2024 consensus_queue.cc:260] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:19.881652 2031 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:19.882315 2031 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:19.882884 2025 ts_tablet_manager.cc:1428] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:44:19.878819 2026 ts_tablet_manager.cc:1428] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:44:19.883522 2043 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:19.883935 2043 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:19.884603 2024 ts_tablet_manager.cc:1428] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 15:44:20.008522 2032 raft_consensus.cc:482] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:20.008886 2032 raft_consensus.cc:504] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:20.010257 2032 leader_election.cc:288] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855), e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371)
I20260929 15:44:20.011014 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e3f1dc729c4ec79ebdbce8399a1878" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" is_pre_election: true
I20260929 15:44:20.011132 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e3f1dc729c4ec79ebdbce8399a1878" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dffe1170614540a8b42bb1b886ddaf" is_pre_election: true
I20260929 15:44:20.011327 1674 raft_consensus.cc:2454] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 0.
I20260929 15:44:20.011615 1955 raft_consensus.cc:2454] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 0.
I20260929 15:44:20.011855 1747 leader_election.cc:302] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a0bd71ed4c6fbee9715607d1ff93, d6b0d1426fed441b9ecb398a3a331813; no voters:
I20260929 15:44:20.012346 2032 raft_consensus.cc:2781] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:20.012477 2032 raft_consensus.cc:482] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:20.012622 2032 raft_consensus.cc:3037] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:20.014982 2032 raft_consensus.cc:504] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:20.015739 2032 leader_election.cc:288] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 election: Requested vote from peers 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855), e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371)
I20260929 15:44:20.016443 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e3f1dc729c4ec79ebdbce8399a1878" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dffe1170614540a8b42bb1b886ddaf"
I20260929 15:44:20.016556 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e3f1dc729c4ec79ebdbce8399a1878" candidate_uuid: "d6b0d1426fed441b9ecb398a3a331813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93"
I20260929 15:44:20.016695 1955 raft_consensus.cc:3037] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:20.017011 1674 raft_consensus.cc:3037] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:20.019181 1955 raft_consensus.cc:2454] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 1.
I20260929 15:44:20.019737 1746 leader_election.cc:302] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b0d1426fed441b9ecb398a3a331813, e6dffe1170614540a8b42bb1b886ddaf; no voters:
I20260929 15:44:20.020104 2032 raft_consensus.cc:2781] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:20.020349 2032 raft_consensus.cc:686] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [term 1 LEADER]: Becoming Leader. State: Replica: d6b0d1426fed441b9ecb398a3a331813, State: Running, Role: LEADER
I20260929 15:44:20.020623 2032 consensus_queue.cc:237] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } }
I20260929 15:44:20.021572 1674 raft_consensus.cc:2454] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b0d1426fed441b9ecb398a3a331813 in term 1.
I20260929 15:44:20.025962 2055 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: TxnStatusTablet state changed. Reason: New leader d6b0d1426fed441b9ecb398a3a331813. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.026374 2055 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:44:20.027427 2068 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:44:20.027691 2068 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:44:20.027796 1517 catalog_manager.cc:5495] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } health_report { overall_health: UNKNOWN } } }
I20260929 15:44:20.103914 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.082526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36685 (local address 127.0.0.1:41912)
0929 15:44:20.082795 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:20.082820 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:20.083009 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:20.096492 (+ 13483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:20.096500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:20.096518 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:20.096758 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:20.096772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.098688 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.098694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:20.100387 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:20.100408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.101527 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.101534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:20.101739 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.101845 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:20.103231 (+ 1386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:20.103267 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:44:20.103332 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 15:44:20.103525 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 15:44:20.105067 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.083050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.0.97.2:36685)
0929 15:44:20.095642 (+ 12592us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.095649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.095686 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:44:20.095830 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.095848 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.095978 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.096946 (+ 968us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:20.097224 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.098511 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.100611 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.101309 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.102050 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.102221 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:20.103089 (+ 868us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:20.104480 (+ 1391us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.104556 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.104653 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12408,"thread_start_us":13209,"threads_started":1}
I20260929 15:44:20.115465 1955 raft_consensus.cc:1261] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Refusing update from remote peer d6b0d1426fed441b9ecb398a3a331813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:20.115690 1674 raft_consensus.cc:1261] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Refusing update from remote peer d6b0d1426fed441b9ecb398a3a331813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:20.117004 2055 consensus_queue.cc:1035] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:20.117542 2032 consensus_queue.cc:1035] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:20.135893 2031 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: TxnStatusTablet state changed. Reason: New leader d6b0d1426fed441b9ecb398a3a331813. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.136384 2031 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:20.141026 2043 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: TxnStatusTablet state changed. Reason: New leader d6b0d1426fed441b9ecb398a3a331813. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.141491 2055 consensus_queue.cc:1035] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:20.141853 2043 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:20.146821 2072 mvcc.cc:205] Tried to move back new op lower bound from 7334693519818194944 to 7334693519447666688. Current Snapshot: MvccSnapshot[applied={T|T < 7334693519818194944}]
I20260929 15:44:20.156548 2074 mvcc.cc:205] Tried to move back new op lower bound from 7334693519818194944 to 7334693519447666688. Current Snapshot: MvccSnapshot[applied={T|T < 7334693519818194944}]
I20260929 15:44:20.167218 2079 consensus_queue.cc:1035] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:20.176705 2043 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.177156 2043 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P e6dffe1170614540a8b42bb1b886ddaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:20.180706 2031 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.181028 2031 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P 2f32a0bd71ed4c6fbee9715607d1ff93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:20.181811 2079 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.182431 2079 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:44:20.192649 2079 tablet_replica.cc:439] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6b0d1426fed441b9ecb398a3a331813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f32a0bd71ed4c6fbee9715607d1ff93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "d6b0d1426fed441b9ecb398a3a331813" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36685 } } peers { permanent_uuid: "e6dffe1170614540a8b42bb1b886ddaf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34371 } } }
I20260929 15:44:20.193066 2079 tablet_replica.cc:442] T 75e3f1dc729c4ec79ebdbce8399a1878 P d6b0d1426fed441b9ecb398a3a331813: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:44:20.209023 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.202824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.0.97.2:36685)
0929 15:44:20.203023 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.203032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.203607 (+ 575us) server_negotiation.cc:407] Connection header received
0929 15:44:20.203847 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.203851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.203964 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.204064 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:20.204682 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.205467 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.207064 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.207525 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.207675 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.208081 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:20.208264 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:20.208372 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.208558 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.208644 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:44:20.236658 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.225747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47245 (local address 127.0.97.2:36685)
0929 15:44:20.226141 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.226150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.229015 (+ 2865us) server_negotiation.cc:407] Connection header received
0929 15:44:20.229249 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.229254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.229328 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.229436 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:20.229970 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.230854 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.233219 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.234008 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.234137 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.235144 (+ 1007us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:20.235386 (+ 242us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:20.235467 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.235899 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.236060 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:44:20.237902 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.225747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36685 (local address 127.0.97.2:47245)
0929 15:44:20.228894 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:20.228914 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:20.229113 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:20.229447 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:20.229452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:20.229466 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:20.229812 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:20.229821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.230994 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.231000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:20.232872 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:20.232905 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.234404 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.234414 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:20.234764 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.234873 (+ 109us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:20.235646 (+ 773us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:20.235660 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:44:20.237114 (+ 1454us) client_negotiation.cc:240] Negotiation successful
0929 15:44:20.237358 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2937,"thread_start_us":2749,"threads_started":1}
I20260929 15:44:20.343466 1516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:20.343745 1516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:20.347581 1516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 337469e10ba949f4bf5f3d03bf9626fd: Sending DeleteTablet for 3 replicas of tablet 8a04455f85024342a88672573cef6ff5
I20260929 15:44:20.348593 1655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a04455f85024342a88672573cef6ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34928
I20260929 15:44:20.348760 1934 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a04455f85024342a88672573cef6ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39222
I20260929 15:44:20.349014 1795 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a04455f85024342a88672573cef6ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41898
I20260929 15:44:20.359851 2093 tablet_replica.cc:330] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: stopping tablet replica
I20260929 15:44:20.360607 2093 raft_consensus.cc:2229] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:20.361074 2093 raft_consensus.cc:2258] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:20.361958 2092 tablet_replica.cc:330] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: stopping tablet replica
I20260929 15:44:20.362346 2092 raft_consensus.cc:2229] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:20.362720 2092 raft_consensus.cc:2258] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:20.362787 2093 ts_tablet_manager.cc:1905] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:20.364869 2092 ts_tablet_manager.cc:1905] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:20.378371 2092 ts_tablet_manager.cc:1918] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:20.378762 2092 log.cc:1196] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/8a04455f85024342a88672573cef6ff5
I20260929 15:44:20.379437 2092 ts_tablet_manager.cc:1939] T 8a04455f85024342a88672573cef6ff5 P 2f32a0bd71ed4c6fbee9715607d1ff93: Deleting consensus metadata
I20260929 15:44:20.382318 2093 ts_tablet_manager.cc:1918] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:20.382527 2093 log.cc:1196] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/8a04455f85024342a88672573cef6ff5
I20260929 15:44:20.382406 1504 catalog_manager.cc:4841] TS 2f32a0bd71ed4c6fbee9715607d1ff93 (127.0.97.1:42855): tablet 8a04455f85024342a88672573cef6ff5 (table test-table [id=68116134aa9a4187963c7c0960bf8dda]) successfully deleted
I20260929 15:44:20.385782 2093 ts_tablet_manager.cc:1939] T 8a04455f85024342a88672573cef6ff5 P e6dffe1170614540a8b42bb1b886ddaf: Deleting consensus metadata
I20260929 15:44:20.387768 1503 catalog_manager.cc:4841] TS e6dffe1170614540a8b42bb1b886ddaf (127.0.97.3:34371): tablet 8a04455f85024342a88672573cef6ff5 (table test-table [id=68116134aa9a4187963c7c0960bf8dda]) successfully deleted
I20260929 15:44:20.391038 2094 tablet_replica.cc:330] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: stopping tablet replica
I20260929 15:44:20.391368 2094 raft_consensus.cc:2229] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:20.392112 2094 raft_consensus.cc:2258] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:20.395673 2094 ts_tablet_manager.cc:1905] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:44:20 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:20.404188 2094 ts_tablet_manager.cc:1918] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:20.404381 2094 log.cc:1196] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/8a04455f85024342a88672573cef6ff5
I20260929 15:44:20.405367 2094 ts_tablet_manager.cc:1939] T 8a04455f85024342a88672573cef6ff5 P d6b0d1426fed441b9ecb398a3a331813: Deleting consensus metadata
I20260929 15:44:20.406769 1504 catalog_manager.cc:4841] TS d6b0d1426fed441b9ecb398a3a331813 (127.0.97.2:36685): tablet 8a04455f85024342a88672573cef6ff5 (table test-table [id=68116134aa9a4187963c7c0960bf8dda]) successfully deleted
I20260929 15:44:20.408875 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.374013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.97.1:42855)
0929 15:44:20.389854 (+ 15841us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.389880 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.389914 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:44:20.390008 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.390013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.390137 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.390318 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:20.392640 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.394073 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.396035 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.396629 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.396928 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.401907 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:20.401925 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:20.401941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:20.401985 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:20.405472 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.406471 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.406479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.406482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.406673 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.407116 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.407126 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.407128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.407524 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:20.407835 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.408245 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.408508 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15667,"thread_start_us":16423,"threads_started":1}
W20260929 15:44:20.410151 1720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34944
I20260929 15:44:20.430346 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.414485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.97.62:33429)
0929 15:44:20.415689 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.415694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.415716 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:44:20.415806 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.415820 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.415890 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.416010 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:20.417945 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.418840 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.420745 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.421223 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.421384 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.423878 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:20.423884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:20.423887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:20.423925 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:20.427150 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.427914 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.427922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.427925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.428067 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.428470 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.428476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.428480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.428760 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:20.429020 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.429696 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.429973 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":926,"threads_started":1}
W20260929 15:44:20.431129 1516 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34468
I20260929 15:44:20.436685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1589
I20260929 15:44:20.456120 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.438011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.97.1:42855)
0929 15:44:20.438299 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.438307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.438819 (+ 512us) server_negotiation.cc:407] Connection header received
0929 15:44:20.439033 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.439039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.439139 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.439297 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:20.440726 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.441847 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.443845 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.444563 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.444703 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.447784 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:20.447791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:20.447796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:20.447845 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:20.451659 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.452776 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.452785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.452790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.453189 (+ 399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.453661 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.453667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.453670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.454057 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:20.454328 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.455233 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.455511 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 15:44:20.456820 1720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34960
W20260929 15:44:20.457554 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1589: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:20.457837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1589
I20260929 15:44:20.484642 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1729
Sep 29 15:44:20 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:20.505623 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.485459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.0.97.2:36685)
0929 15:44:20.485665 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.485669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.485689 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:44:20.485876 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.485880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.485943 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.486035 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:20.487823 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.489251 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.491213 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.491748 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.491871 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.498580 (+ 6709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:20.498598 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:20.498614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:20.498667 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:20.502306 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.503233 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.503242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.503244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.503395 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.503815 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.503820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.503836 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.504316 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:20.504692 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.505011 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.505255 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 15:44:20.506518 1860 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41918
W20260929 15:44:20.507429 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1729: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:20.507611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1729
I20260929 15:44:20.533121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1869
Sep 29 15:44:20 dist-test-slave-7491 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:20.552218 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:20.534081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.0.97.3:34371)
0929 15:44:20.534624 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:20.534629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:20.534646 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:44:20.534721 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:20.534725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:20.534809 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:20.534957 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:20.536888 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.537697 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.539368 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:20.539804 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:20.539923 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:20.544791 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:20.544810 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:20.544828 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:20.544878 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:20.548297 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.549294 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.549304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.549307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.549468 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:20.549992 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:20.549999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:20.550001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:20.550594 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:20.551002 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:20.551497 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 15:44:20.551766 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":337,"threads_started":1}
W20260929 15:44:20.553176 2000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39236
W20260929 15:44:20.554142 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:20.554407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1869
I20260929 15:44:20.579461 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 1486
I20260929 15:44:20.581058 1558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:20.717278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 1486
2026-09-29T15:44:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5622 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:20 dist-test-slave-7491 krb5kdc[2106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:20 dist-test-slave-7491 krb5kdc[2106](info): set up 2 sockets
Sep 29 15:44:20 dist-test-slave-7491 krb5kdc[2106](info): commencing operation
krb5kdc: starting...
W20260929 15:44:21.865012 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:22 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:22.090543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44185
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:22.482028 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:22.482385 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:22.482476 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:22.491120 2119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:22.491336 2119 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:44:22.491385 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:22.491448 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:22.491506 2119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:22.491581 2119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:22.504493 2119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:22.506090 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:22.507755 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:22.524998 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:22.525036 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:22.525059 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:22.525981 2119 server_base.cc:1034] running on GCE node
I20260929 15:44:22.526889 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:22.528662 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:22.529937 2119 hybrid_clock.cc:648] HybridClock initialized: now 1790696662529842 us; error 75 us; skew 500 ppm
I20260929 15:44:22.530308 2119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:22 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:22.541383 2119 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:22.543722 2119 webserver.cc:460] Webserver started at http://127.0.97.62:43505/ using document root <none> and password file <none>
I20260929 15:44:22.544867 2119 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:22.544992 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:22.545353 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:22.549255 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "eb7c1c66915e45f4a27c4453d0d7a41b"
format_stamp: "Formatted at 2026-09-29 15:44:22 on dist-test-slave-7491"
server_key: "471088159fa8c4c25e5395d7ff8137fc"
server_key_iv: "fb9592ea22cb7eb31539792d73eb8258"
server_key_version: "encryptionkey@0"
I20260929 15:44:22.550248 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb7c1c66915e45f4a27c4453d0d7a41b"
format_stamp: "Formatted at 2026-09-29 15:44:22 on dist-test-slave-7491"
server_key: "471088159fa8c4c25e5395d7ff8137fc"
server_key_iv: "fb9592ea22cb7eb31539792d73eb8258"
server_key_version: "encryptionkey@0"
I20260929 15:44:22.557672 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 15:44:22.564342 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:22.566905 2119 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260929 15:44:22.567219 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "eb7c1c66915e45f4a27c4453d0d7a41b"
format_stamp: "Formatted at 2026-09-29 15:44:22 on dist-test-slave-7491"
server_key: "471088159fa8c4c25e5395d7ff8137fc"
server_key_iv: "fb9592ea22cb7eb31539792d73eb8258"
server_key_version: "encryptionkey@0"
I20260929 15:44:22.567512 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:22.597558 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:22.603392 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:22.603660 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:22.631917 2119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44185
I20260929 15:44:22.631944 2195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44185 every 8 connection(s)
I20260929 15:44:22.634085 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:22.636361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 2119
I20260929 15:44:22.636600 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:22.637458 388 external_mini_cluster.cc:1428] Setting key 6d3aa23fb582eee87479bffdd5ab1dd6
I20260929 15:44:22.639755 2196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:44:22 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:22.653615 2196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Bootstrap starting.
I20260929 15:44:22.657778 2196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:22.659394 2196 log.cc:824] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:22.664029 2196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: No bootstrap required, opened a new log
I20260929 15:44:22.666308 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:22.639906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.0.97.62:44185)
0929 15:44:22.640802 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:22.640816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:22.640862 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:44:22.642608 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:22.642631 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:22.643070 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:22.643524 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:22.645138 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:22.646564 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:22.647872 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:22.648687 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:22.648897 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:22.654668 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:22.654692 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:22.654719 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:22.654786 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:22.660874 (+ 6088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:22.662291 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:22.662301 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:22.662309 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:22.662534 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:22.663045 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:22.663050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:22.663051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:22.663641 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:22.663890 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:22.664685 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 15:44:22.665158 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":432,"threads_started":1}
I20260929 15:44:22.675118 2196 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44185 } }
I20260929 15:44:22.675729 2196 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:22.675834 2196 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7c1c66915e45f4a27c4453d0d7a41b, State: Initialized, Role: FOLLOWER
I20260929 15:44:22.676605 2196 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44185 } }
I20260929 15:44:22.676846 2196 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:22.677021 2196 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:22.677210 2196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:22.759349 2196 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44185 } }
I20260929 15:44:22.760285 2196 leader_election.cc:302] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7c1c66915e45f4a27c4453d0d7a41b; no voters:
I20260929 15:44:22.761130 2196 leader_election.cc:288] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:22.761301 2201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:22.762826 2201 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [term 1 LEADER]: Becoming Leader. State: Replica: eb7c1c66915e45f4a27c4453d0d7a41b, State: Running, Role: LEADER
I20260929 15:44:22.763648 2201 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44185 } }
I20260929 15:44:22.764442 2196 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:22.767694 2202 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44185 } } }
I20260929 15:44:22.768301 2202 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:22.768926 2203 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7c1c66915e45f4a27c4453d0d7a41b. Latest consensus state: current_term: 1 leader_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c1c66915e45f4a27c4453d0d7a41b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44185 } } }
I20260929 15:44:22.769333 2203 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:22.772385 2208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:22.777585 2208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:22.794916 2208 catalog_manager.cc:1347] Generated new cluster ID: 216ff2fa60d54aa1a2038de80dac3844
I20260929 15:44:22.795083 2208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:22.807787 2208 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:22.809927 2208 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:22.820858 2208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Generated new TSK 0
I20260929 15:44:22.821949 2208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260929 15:44:23.224529 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.384s user 0.007s sys 0.007s
I20260929 15:44:23.225689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44185
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:23.623835 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:23.624297 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:23.624491 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:23.632743 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:23.632951 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:23.633129 2222 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:44:23.633212 2222 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:44:23.633363 2222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:23.646319 2222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:23.647837 2222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:23.649430 2222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:23.666435 2230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:23.666484 2227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:23.666435 2228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:23.667338 2222 server_base.cc:1034] running on GCE node
I20260929 15:44:23.668248 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:23.669550 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:23.670957 2222 hybrid_clock.cc:648] HybridClock initialized: now 1790696663670880 us; error 60 us; skew 500 ppm
I20260929 15:44:23.671339 2222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:23 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:23.677764 2222 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:23.680239 2222 webserver.cc:460] Webserver started at http://127.0.97.1:41525/ using document root <none> and password file <none>
I20260929 15:44:23.681072 2222 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:23.681216 2222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:23.681514 2222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:23.685006 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "5f484e8e8a37417581f8a931e34e5594"
format_stamp: "Formatted at 2026-09-29 15:44:23 on dist-test-slave-7491"
server_key: "bb32ddb21154c4b3608d44d7a0633258"
server_key_iv: "6c12a372940b78b80debef215a0e4963"
server_key_version: "encryptionkey@0"
I20260929 15:44:23.685931 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5f484e8e8a37417581f8a931e34e5594"
format_stamp: "Formatted at 2026-09-29 15:44:23 on dist-test-slave-7491"
server_key: "bb32ddb21154c4b3608d44d7a0633258"
server_key_iv: "6c12a372940b78b80debef215a0e4963"
server_key_version: "encryptionkey@0"
I20260929 15:44:23.692472 2222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:44:23.697297 2237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:23.699471 2222 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:44:23.699707 2222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5f484e8e8a37417581f8a931e34e5594"
format_stamp: "Formatted at 2026-09-29 15:44:23 on dist-test-slave-7491"
server_key: "bb32ddb21154c4b3608d44d7a0633258"
server_key_iv: "6c12a372940b78b80debef215a0e4963"
server_key_version: "encryptionkey@0"
I20260929 15:44:23.700124 2222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:23.722888 2222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:23.728091 2222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:23.728411 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:23.731448 2222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:23.731618 2222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:23.732012 2222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:23.732511 2222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 15:44:23 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:23.765470 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:23.732434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45511 (local address 127.0.97.62:44185)
0929 15:44:23.733146 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:23.733156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:23.734628 (+ 1472us) server_negotiation.cc:407] Connection header received
0929 15:44:23.738877 (+ 4249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:23.738886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:23.739014 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:23.739191 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:23.742025 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.743540 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.747531 (+ 3991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.748208 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.748384 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:23.756817 (+ 8433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:23.756825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:23.756830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:23.756885 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:23.760854 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:23.762172 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:23.762181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:23.762186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:23.762389 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:23.763009 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:23.763015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:23.763019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:23.763343 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:23.763549 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:23.764781 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0929 15:44:23.765043 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":412,"threads_started":1}
I20260929 15:44:23.767150 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:23.733327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.97.1:45511)
0929 15:44:23.734393 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:23.734459 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:23.738066 (+ 3607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:23.740130 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:23.740142 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:23.740990 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:23.741758 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:23.741781 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.743878 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.743891 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:23.747144 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:23.747171 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.748542 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.748554 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:23.749043 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:23.750717 (+ 1674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:23.750765 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:23.756295 (+ 5530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:23.761122 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:23.761132 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:23.761150 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:23.761969 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:23.762555 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:23.762561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:23.762571 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:23.762855 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:23.763539 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:23.763548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:23.764132 (+ 584us) client_negotiation.cc:769] Sending connection context
0929 15:44:23.764648 (+ 516us) client_negotiation.cc:240] Negotiation successful
0929 15:44:23.765261 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":353,"threads_started":1}
I20260929 15:44:23.793599 2222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40193
I20260929 15:44:23.793637 2357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40193 every 8 connection(s)
I20260929 15:44:23.795516 2222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:23.799045 2358 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44185
I20260929 15:44:23.799357 2358 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:23.799526 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:23.774924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33261 (local address 127.0.97.62:44185)
0929 15:44:23.775160 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:23.775169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:23.776749 (+ 1580us) server_negotiation.cc:407] Connection header received
0929 15:44:23.778397 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:23.778408 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:23.778609 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:23.778852 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:23.780926 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.782075 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.784719 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.785752 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.785946 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:23.789343 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:23.789355 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:23.789363 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:23.789427 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:23.793711 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:23.795032 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:23.795063 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:23.795066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:23.795329 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:23.796515 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:23.796521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:23.796524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:23.797048 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:23.797249 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:23.798498 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0929 15:44:23.798776 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 15:44:23.799576 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:23.775471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.97.1:33261)
0929 15:44:23.776572 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:23.776616 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:23.776799 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:23.779093 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:23.779101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:23.780034 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:23.780666 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:23.780688 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.782274 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.782283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:23.784345 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:23.784363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:23.786056 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:23.786070 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:23.786341 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:23.787306 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:23.787338 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:23.788980 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:23.793937 (+ 4957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:23.793944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:23.793949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:23.794566 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:23.795529 (+ 963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:23.795538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:23.795545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:23.795982 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:23.797270 (+ 1288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:23.797281 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:23.797688 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 15:44:23.798385 (+ 697us) client_negotiation.cc:240] Negotiation successful
0929 15:44:23.798994 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":557,"threads_started":1}
I20260929 15:44:23.800446 2358 heartbeater.cc:511] Master 127.0.97.62:44185 requested a full tablet report, sending...
I20260929 15:44:23.801100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 2222
I20260929 15:44:23.801357 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:23.802140 388 external_mini_cluster.cc:1428] Setting key 9118f7983b7eee994aa76efd8a491872
I20260929 15:44:23.802766 2151 ts_manager.cc:194] Registered new tserver with Master: 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193)
I20260929 15:44:23.805193 2151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45511
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:23.852645 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44185
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:24.264243 2362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:24.264663 2362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:24.264856 2362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:24.274535 2362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:24.274748 2362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:24.274915 2362 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:44:24.275008 2362 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:44:24.275158 2362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:24.287012 2362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:24.288546 2362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:24.290050 2362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:24.306484 2370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:24.306448 2368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:24.306512 2367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:24.307657 2362 server_base.cc:1034] running on GCE node
I20260929 15:44:24.308454 2362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:24.309545 2362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:24.310786 2362 hybrid_clock.cc:648] HybridClock initialized: now 1790696664310715 us; error 68 us; skew 500 ppm
I20260929 15:44:24.311184 2362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:24 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:24.315279 2362 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:24.317698 2362 webserver.cc:460] Webserver started at http://127.0.97.2:44139/ using document root <none> and password file <none>
I20260929 15:44:24.318796 2362 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:24.319005 2362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:24.319430 2362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:24.323742 2362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "12d21183f2464a6593c8c4e0bf0295ce"
format_stamp: "Formatted at 2026-09-29 15:44:24 on dist-test-slave-7491"
server_key: "410df35f0e8d372f6172a8c75c91b9d4"
server_key_iv: "9acc7ec81c1e44d332226ed5d85e479a"
server_key_version: "encryptionkey@0"
I20260929 15:44:24.324931 2362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "12d21183f2464a6593c8c4e0bf0295ce"
format_stamp: "Formatted at 2026-09-29 15:44:24 on dist-test-slave-7491"
server_key: "410df35f0e8d372f6172a8c75c91b9d4"
server_key_iv: "9acc7ec81c1e44d332226ed5d85e479a"
server_key_version: "encryptionkey@0"
I20260929 15:44:24.332335 2362 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 15:44:24.336606 2377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:24.338498 2362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:44:24.338850 2362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "12d21183f2464a6593c8c4e0bf0295ce"
format_stamp: "Formatted at 2026-09-29 15:44:24 on dist-test-slave-7491"
server_key: "410df35f0e8d372f6172a8c75c91b9d4"
server_key_iv: "9acc7ec81c1e44d332226ed5d85e479a"
server_key_version: "encryptionkey@0"
I20260929 15:44:24.339196 2362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:24.367378 2362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:24.373710 2362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:24.374070 2362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:24.377269 2362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:24.377426 2362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:24.377593 2362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:24.377710 2362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:44:24 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:24.407578 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:24.378197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45567 (local address 127.0.97.62:44185)
0929 15:44:24.378709 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:24.378716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:24.380277 (+ 1561us) server_negotiation.cc:407] Connection header received
0929 15:44:24.383144 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:24.383150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:24.383262 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:24.383394 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:24.386551 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.387600 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.390642 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.391416 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.391593 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:24.399598 (+ 8005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:24.399610 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:24.399618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:24.399687 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:24.403885 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:24.404984 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:24.404990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:24.404993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:24.405151 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:24.405642 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:24.405647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:24.405651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:24.405954 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:24.406133 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:24.406912 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 15:44:24.407181 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":268,"threads_started":1}
I20260929 15:44:24.408922 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:24.378999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.97.2:45567)
0929 15:44:24.379976 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:24.380037 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:24.382652 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:24.383797 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:24.383813 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:24.384985 (+ 1172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:24.386255 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:24.386292 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.387804 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.387809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:24.390023 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:24.390043 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.391817 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.391831 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:24.392929 (+ 1098us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:24.394320 (+ 1391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:24.394505 (+ 185us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:24.399136 (+ 4631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:24.404209 (+ 5073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:24.404225 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:24.404253 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:24.404824 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:24.405278 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:24.405283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:24.405289 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:24.405510 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:24.406095 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:24.406104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:24.406514 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 15:44:24.406818 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 15:44:24.407411 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":419,"threads_started":1}
I20260929 15:44:24.437287 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:24.417493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.97.2:52661)
0929 15:44:24.418251 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:24.418275 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:24.418453 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:24.419004 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:24.419007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:24.419557 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:24.420318 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:24.420350 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.422048 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.422053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:24.423543 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:24.423560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.424769 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.424782 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:24.424990 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:24.425838 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:24.425858 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:24.426898 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:24.433262 (+ 6364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:24.433267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:24.433271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:24.433789 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:24.434494 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:24.434499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:24.434501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:24.434677 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:24.435690 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:24.435696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:24.435947 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 15:44:24.436414 (+ 467us) client_negotiation.cc:240] Negotiation successful
0929 15:44:24.436909 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":408,"threads_started":1}
I20260929 15:44:24.437295 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:24.417384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52661 (local address 127.0.97.62:44185)
0929 15:44:24.417706 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:24.417717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:24.418401 (+ 684us) server_negotiation.cc:407] Connection header received
0929 15:44:24.418652 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:24.418662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:24.418787 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:24.418950 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:24.420657 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.421824 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.423775 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.424509 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.424680 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:24.427323 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:24.427336 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:24.427345 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:24.427441 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:24.432954 (+ 5513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:24.434049 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:24.434057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:24.434064 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:24.434301 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:24.434837 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:24.434845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:24.434852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:24.435463 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:24.435684 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:24.436481 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 15:44:24.436806 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:44:24.443819 2362 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35501
I20260929 15:44:24.443931 2497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35501 every 8 connection(s)
I20260929 15:44:24.445485 2362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:24.448666 2498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44185
I20260929 15:44:24.449007 2498 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:24.449584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 2362
I20260929 15:44:24.449795 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:24.449934 2498 heartbeater.cc:511] Master 127.0.97.62:44185 requested a full tablet report, sending...
I20260929 15:44:24.450433 388 external_mini_cluster.cc:1428] Setting key 6b27d97524a71d054b5882ed76bb93fe
I20260929 15:44:24.452211 2150 ts_manager.cc:194] Registered new tserver with Master: 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501)
I20260929 15:44:24.453534 2150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45567
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:24.500418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44185
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:44:24.808727 2358 heartbeater.cc:503] Master 127.0.97.62:44185 was elected leader, sending a full tablet report...
W20260929 15:44:24.895140 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:24.895483 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:24.895593 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:24.903164 2502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:24.903326 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:24.903415 2502 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:44:24.903468 2502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:44:24.903541 2502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:24.914388 2502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:24.915866 2502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:24.917299 2502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:24.932678 2507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:24.932690 2508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:24.933457 2510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:24.933970 2502 server_base.cc:1034] running on GCE node
I20260929 15:44:24.934705 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:24.935803 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:24.937105 2502 hybrid_clock.cc:648] HybridClock initialized: now 1790696664937015 us; error 55 us; skew 500 ppm
I20260929 15:44:24.937476 2502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:24 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:24.941917 2502 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:24.943892 2502 webserver.cc:460] Webserver started at http://127.0.97.3:41027/ using document root <none> and password file <none>
I20260929 15:44:24.944842 2502 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:24.944965 2502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:24.945310 2502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:24.948738 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "b0382bc50fab4883a36c949b5ee7f926"
format_stamp: "Formatted at 2026-09-29 15:44:24 on dist-test-slave-7491"
server_key: "71ac34ceef4cf2d2ca80367e8a00ebfd"
server_key_iv: "780b0d60fdaa2b20e51df4d784438855"
server_key_version: "encryptionkey@0"
I20260929 15:44:24.949667 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b0382bc50fab4883a36c949b5ee7f926"
format_stamp: "Formatted at 2026-09-29 15:44:24 on dist-test-slave-7491"
server_key: "71ac34ceef4cf2d2ca80367e8a00ebfd"
server_key_iv: "780b0d60fdaa2b20e51df4d784438855"
server_key_version: "encryptionkey@0"
I20260929 15:44:24.956118 2502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 15:44:24.960793 2517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:24.962549 2502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:44:24.962810 2502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "b0382bc50fab4883a36c949b5ee7f926"
format_stamp: "Formatted at 2026-09-29 15:44:24 on dist-test-slave-7491"
server_key: "71ac34ceef4cf2d2ca80367e8a00ebfd"
server_key_iv: "780b0d60fdaa2b20e51df4d784438855"
server_key_version: "encryptionkey@0"
I20260929 15:44:24.963056 2502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:24.977113 2502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:24.982299 2502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:24.982649 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:24.986601 2502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:24.986886 2502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:44:24.987054 2502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:24.987174 2502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:44:24 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:25.015153 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:24.986457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33567 (local address 127.0.97.62:44185)
0929 15:44:24.987057 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:24.987064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:24.988468 (+ 1404us) server_negotiation.cc:407] Connection header received
0929 15:44:24.991039 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:24.991046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:24.991184 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:24.991339 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:24.994492 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.995665 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.998462 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.999122 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.999300 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.005186 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:25.005192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:25.005196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:25.005236 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:25.009140 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.010971 (+ 1831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.010977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.010980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.011217 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.012061 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.012069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.012073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.012617 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:25.012786 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.014354 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.014683 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":329,"threads_started":1}
I20260929 15:44:25.016920 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:24.987089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.97.3:33567)
0929 15:44:24.988254 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:24.988312 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:24.990531 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:24.991854 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:24.991875 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:24.992903 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:24.994187 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:24.994231 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.995885 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.995898 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:24.998263 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:24.998280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:24.999673 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:24.999678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.000043 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.001044 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:25.001070 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:25.004835 (+ 3765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:25.009381 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:25.009393 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:25.009412 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:25.010638 (+ 1226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:25.011355 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:25.011367 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:25.011371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:25.011875 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:25.012941 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:25.012953 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:25.013815 (+ 862us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.014224 (+ 409us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.014963 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":458,"threads_started":1}
I20260929 15:44:25.044148 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.025389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39285 (local address 127.0.97.62:44185)
0929 15:44:25.025582 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.025586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.026313 (+ 727us) server_negotiation.cc:407] Connection header received
0929 15:44:25.026548 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.026553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.026629 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.026745 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:25.027746 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.028575 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.031545 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.032117 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.032399 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.035587 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:25.035594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:25.035598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:25.035639 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:25.039849 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.041342 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.041354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.041359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.041556 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.042136 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.042140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.042142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.042458 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:25.042596 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.043353 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.043703 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:44:25.044970 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.025520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.97.3:39285)
0929 15:44:25.026185 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.026203 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.026396 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.026771 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.026775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.027288 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:25.027586 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.027596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.028759 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.028765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.031249 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.031273 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.032474 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.032487 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.032799 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.033724 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:25.033794 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:25.035253 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:25.040104 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:25.040117 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:25.040123 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:25.041113 (+ 990us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:25.041727 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:25.041735 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:25.041741 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:25.041962 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:25.042607 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:25.042623 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:25.042906 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.043302 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.044098 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":379,"threads_started":1}
I20260929 15:44:25.048103 2502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45305
I20260929 15:44:25.048130 2637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45305 every 8 connection(s)
I20260929 15:44:25.049852 2502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:25.052536 2638 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44185
I20260929 15:44:25.052925 2638 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:25.053859 2638 heartbeater.cc:511] Master 127.0.97.62:44185 requested a full tablet report, sending...
I20260929 15:44:25.054590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 2502
I20260929 15:44:25.054816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:25.055470 388 external_mini_cluster.cc:1428] Setting key 5b861ee4c566d8f8e0aa1c54a02ac1d7
I20260929 15:44:25.056015 2150 ts_manager.cc:194] Registered new tserver with Master: b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305)
I20260929 15:44:25.057114 2150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33567
I20260929 15:44:25.058715 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:44:25 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:44:25 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:25.115556 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.098411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.0.97.62:44185)
0929 15:44:25.098665 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.098670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.098698 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:44:25.098887 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.098891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.098960 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.099064 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:25.100868 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.101645 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.103425 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.104139 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.104364 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.108970 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:25.108976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:25.108980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:25.109024 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:25.112640 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.113490 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.113497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.113501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.113650 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.114039 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.114045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.114049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.114364 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:25.114640 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.114962 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.115216 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 15:44:25.120550 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:25.128365 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:25.166527 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.143168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.0.1:50932)
0929 15:44:25.150343 (+ 7175us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.150423 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.150719 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.157847 (+ 7128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.157854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.157881 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.158839 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.158854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.160832 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.160838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.163578 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.163603 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.165283 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.165289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.165607 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.165660 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.165833 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.166069 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6746,"mutex_wait_us":9,"thread_start_us":450,"threads_started":1}
I20260929 15:44:25.167450 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.143366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.0.97.2:35501)
0929 15:44:25.157212 (+ 13846us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.157222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.157256 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:44:25.157362 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.157369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.157605 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.158097 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.159063 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.160675 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.163793 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.165055 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.165224 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.165478 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.166826 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.167125 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13640,"thread_start_us":13536,"threads_started":1}
I20260929 15:44:25.168018 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.143620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40193 (local address 127.0.0.1:60586)
0929 15:44:25.156139 (+ 12519us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.156193 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.156385 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.157644 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.157656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.157702 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.159064 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.159085 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.161191 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.161199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.163874 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.163889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.166310 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.166318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.166487 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.166524 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.166633 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.167646 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12305,"mutex_wait_us":31,"thread_start_us":12866,"threads_started":1}
I20260929 15:44:25.168325 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.144115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.0.97.1:40193)
0929 15:44:25.156347 (+ 12232us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.156358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.156394 (+ 36us) server_negotiation.cc:407] Connection header received
0929 15:44:25.156745 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.156755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.157090 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.157359 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.159362 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.161039 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.164088 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.165818 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.165960 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.166149 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.166942 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.167151 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12011,"thread_start_us":12240,"threads_started":1}
I20260929 15:44:25.168460 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.144820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.0.1:36904)
0929 15:44:25.159188 (+ 14368us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.159218 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.159362 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.160120 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.160126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.160159 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.160748 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.160767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.162858 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.162864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.165151 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.165168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.167592 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.167602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.167825 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.167885 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.167970 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.168107 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12975,"thread_start_us":12728,"threads_started":1}
I20260929 15:44:25.169193 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.145622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.0.97.3:45305)
0929 15:44:25.158298 (+ 12676us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.158310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.159516 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 15:44:25.159634 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.159642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.159876 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.160094 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.160994 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.162666 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.165376 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.167057 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.167192 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.167406 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.168731 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.168889 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12458,"thread_start_us":12569,"threads_started":1}
I20260929 15:44:25.172813 2428 tablet_service.cc:1467] Processing CreateTablet for tablet d4ec2d70c4f14943a1f5d5d7a41b676a (DEFAULT_TABLE table=test-table [id=5fc991670cb44c59b1292c89fb9258cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:25.172811 2288 tablet_service.cc:1467] Processing CreateTablet for tablet d4ec2d70c4f14943a1f5d5d7a41b676a (DEFAULT_TABLE table=test-table [id=5fc991670cb44c59b1292c89fb9258cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:25.174746 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ec2d70c4f14943a1f5d5d7a41b676a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.174764 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ec2d70c4f14943a1f5d5d7a41b676a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.174268 2568 tablet_service.cc:1467] Processing CreateTablet for tablet d4ec2d70c4f14943a1f5d5d7a41b676a (DEFAULT_TABLE table=test-table [id=5fc991670cb44c59b1292c89fb9258cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:25.175834 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ec2d70c4f14943a1f5d5d7a41b676a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.188988 2657 tablet_bootstrap.cc:492] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Bootstrap starting.
I20260929 15:44:25.193262 2658 tablet_bootstrap.cc:492] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Bootstrap starting.
I20260929 15:44:25.194893 2659 tablet_bootstrap.cc:492] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Bootstrap starting.
I20260929 15:44:25.195839 2657 tablet_bootstrap.cc:654] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.197932 2657 log.cc:824] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:25.198724 2658 tablet_bootstrap.cc:654] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.199239 2659 tablet_bootstrap.cc:654] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.200280 2658 log.cc:824] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:25.200659 2659 log.cc:824] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:25.202807 2657 tablet_bootstrap.cc:492] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: No bootstrap required, opened a new log
I20260929 15:44:25.203156 2657 ts_tablet_manager.cc:1397] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260929 15:44:25.203377 2658 tablet_bootstrap.cc:492] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: No bootstrap required, opened a new log
I20260929 15:44:25.203655 2658 ts_tablet_manager.cc:1397] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 15:44:25.203833 2659 tablet_bootstrap.cc:492] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: No bootstrap required, opened a new log
I20260929 15:44:25.204087 2659 ts_tablet_manager.cc:1397] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 15:44:25.212055 2659 raft_consensus.cc:348] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.212726 2659 raft_consensus.cc:374] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.212837 2659 raft_consensus.cc:729] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.213497 2659 consensus_queue.cc:260] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.214982 2498 heartbeater.cc:503] Master 127.0.97.62:44185 was elected leader, sending a full tablet report...
I20260929 15:44:25.214756 2658 raft_consensus.cc:348] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.214756 2657 raft_consensus.cc:348] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.215209 2658 raft_consensus.cc:374] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.215209 2657 raft_consensus.cc:374] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.215306 2658 raft_consensus.cc:729] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f484e8e8a37417581f8a931e34e5594, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.215310 2657 raft_consensus.cc:729] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.215960 2659 ts_tablet_manager.cc:1428] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260929 15:44:25.215989 2658 consensus_queue.cc:260] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.215989 2657 consensus_queue.cc:260] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.218019 2638 heartbeater.cc:503] Master 127.0.97.62:44185 was elected leader, sending a full tablet report...
I20260929 15:44:25.218384 2658 ts_tablet_manager.cc:1428] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 15:44:25.218468 2657 ts_tablet_manager.cc:1428] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260929 15:44:25.293599 2663 raft_consensus.cc:482] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:25.293990 2663 raft_consensus.cc:504] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.296651 2663 leader_election.cc:288] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305), 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193)
W20260929 15:44:25.299759 2359 tablet.cc:2369] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:44:25.301601 2639 tablet.cc:2369] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:25.306923 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.297399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58119 (local address 127.0.97.1:40193)
0929 15:44:25.297620 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.297627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.298620 (+ 993us) server_negotiation.cc:407] Connection header received
0929 15:44:25.298880 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.298886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.298984 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.299105 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.299989 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.301160 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.304198 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.305622 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.305779 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.305910 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.306472 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.306638 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 15:44:25.306923 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.297336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44977 (local address 127.0.97.3:45305)
0929 15:44:25.297610 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.297617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.298689 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 15:44:25.298910 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.298915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.298995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.299109 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.299989 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.301159 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.304198 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.305594 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.305779 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.305910 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.306472 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.306624 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 15:44:25.307097 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.297455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40193 (local address 127.0.97.2:58119)
0929 15:44:25.298441 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.298473 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.298702 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.299181 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.299186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.299219 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.299805 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.299813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.301368 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.301374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.303951 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.303965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.305959 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.305967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.306208 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.306279 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.306398 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.306586 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":77,"thread_start_us":433,"threads_started":1}
I20260929 15:44:25.307145 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.297467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.97.2:44977)
0929 15:44:25.298594 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.298612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.298771 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.299181 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.299186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.299245 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.299795 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.299810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.301368 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.301374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.303844 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.303866 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.305959 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.305967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.306222 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.306289 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.306398 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.306580 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":163,"spinlock_wait_cycles":2944,"thread_start_us":873,"threads_started":1}
I20260929 15:44:25.308688 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ec2d70c4f14943a1f5d5d7a41b676a" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0382bc50fab4883a36c949b5ee7f926" is_pre_election: true
I20260929 15:44:25.308686 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ec2d70c4f14943a1f5d5d7a41b676a" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594" is_pre_election: true
I20260929 15:44:25.309370 2588 raft_consensus.cc:2454] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 0.
I20260929 15:44:25.309370 2308 raft_consensus.cc:2454] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 0.
I20260929 15:44:25.310458 2378 leader_election.cc:302] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, 5f484e8e8a37417581f8a931e34e5594, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:25.311069 2663 raft_consensus.cc:2781] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:25.311260 2663 raft_consensus.cc:482] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:25.311548 2663 raft_consensus.cc:3037] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.314071 2663 raft_consensus.cc:504] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.314818 2663 leader_election.cc:288] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 election: Requested vote from peers b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305), 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193)
I20260929 15:44:25.315548 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ec2d70c4f14943a1f5d5d7a41b676a" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0382bc50fab4883a36c949b5ee7f926"
I20260929 15:44:25.315548 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ec2d70c4f14943a1f5d5d7a41b676a" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594"
I20260929 15:44:25.315797 2308 raft_consensus.cc:3037] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.315797 2588 raft_consensus.cc:3037] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.319370 2588 raft_consensus.cc:2454] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 1.
I20260929 15:44:25.319370 2308 raft_consensus.cc:2454] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 1.
I20260929 15:44:25.320016 2378 leader_election.cc:302] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, 5f484e8e8a37417581f8a931e34e5594, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:25.320451 2663 raft_consensus.cc:2781] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:25.320936 2663 raft_consensus.cc:686] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 1 LEADER]: Becoming Leader. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Running, Role: LEADER
I20260929 15:44:25.321485 2663 consensus_queue.cc:237] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } }
I20260929 15:44:25.327103 2150 catalog_manager.cc:5495] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "12d21183f2464a6593c8c4e0bf0295ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } health_report { overall_health: UNKNOWN } } }
Sep 29 15:44:25 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:25.403609 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.387201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44185 (local address 127.0.0.1:52258)
0929 15:44:25.388021 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.388040 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.388324 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.388747 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.388750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.389516 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:25.389766 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.389775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.390924 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.390929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.391930 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.391941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.392980 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.392989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.393165 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.394137 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:25.394162 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:25.397121 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:25.400589 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:25.400607 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:25.400614 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:25.401110 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:25.401690 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:25.401693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:25.401696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:25.401843 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:25.402554 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:25.402562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:25.402781 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.402950 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.403222 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":475,"threads_started":1}
I20260929 15:44:25.403939 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.387179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.0.97.62:44185)
0929 15:44:25.387395 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.387399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.388281 (+ 882us) server_negotiation.cc:407] Connection header received
0929 15:44:25.388473 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.388477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.388540 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.388641 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:25.389998 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.390774 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.392129 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.392657 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.392781 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.397464 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:25.397469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:25.397473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:25.397512 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:25.400395 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.401313 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.401321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.401324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.401530 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:25.402014 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:25.402018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:25.402020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:25.402383 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:25.402535 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.403118 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.403522 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:44:25.410789 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:44:25.412240 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:25.421761 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1d55a6e19b4be08bf2ab6adcebe498 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72e86edc6efd4c37be31a8bf1d75dfed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:44:25.421545 2288 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1d55a6e19b4be08bf2ab6adcebe498 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72e86edc6efd4c37be31a8bf1d75dfed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:44:25.423066 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1d55a6e19b4be08bf2ab6adcebe498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.423133 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1d55a6e19b4be08bf2ab6adcebe498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.422405 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1d55a6e19b4be08bf2ab6adcebe498 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72e86edc6efd4c37be31a8bf1d75dfed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:44:25.423561 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1d55a6e19b4be08bf2ab6adcebe498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.429555 2659 tablet_bootstrap.cc:492] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: Bootstrap starting.
I20260929 15:44:25.430974 2658 tablet_bootstrap.cc:492] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: Bootstrap starting.
I20260929 15:44:25.430989 2657 tablet_bootstrap.cc:492] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: Bootstrap starting.
I20260929 15:44:25.431174 2659 tablet_bootstrap.cc:654] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.432451 2658 tablet_bootstrap.cc:654] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.432797 2657 tablet_bootstrap.cc:654] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.433972 2659 tablet_bootstrap.cc:492] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: No bootstrap required, opened a new log
I20260929 15:44:25.434134 2659 ts_tablet_manager.cc:1397] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 15:44:25.435019 2659 raft_consensus.cc:348] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.435225 2657 tablet_bootstrap.cc:492] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: No bootstrap required, opened a new log
I20260929 15:44:25.435380 2659 raft_consensus.cc:374] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.435494 2659 raft_consensus.cc:729] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.435494 2657 ts_tablet_manager.cc:1397] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:44:25.435848 2658 tablet_bootstrap.cc:492] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: No bootstrap required, opened a new log
I20260929 15:44:25.435770 2659 consensus_queue.cc:260] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.436468 2659 ts_tablet_manager.cc:1428] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260929 15:44:25.436403 2663 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.436575 2657 raft_consensus.cc:348] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.436810 2663 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.436960 2657 raft_consensus.cc:374] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.437117 2657 raft_consensus.cc:729] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.437134 2658 ts_tablet_manager.cc:1397] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 15:44:25.437415 2657 consensus_queue.cc:260] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.438004 2658 raft_consensus.cc:348] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.437932 2664 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.438246 2658 raft_consensus.cc:374] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.438340 2664 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.438331 2658 raft_consensus.cc:729] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f484e8e8a37417581f8a931e34e5594, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.438786 2658 consensus_queue.cc:260] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.439270 2665 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.439663 2665 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.440299 2657 ts_tablet_manager.cc:1428] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260929 15:44:25.441059 2658 ts_tablet_manager.cc:1428] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260929 15:44:25.448247 2499 tablet.cc:2369] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:25.516431 2664 raft_consensus.cc:482] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:25.516675 2664 raft_consensus.cc:504] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.518566 2664 leader_election.cc:288] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193), 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501)
I20260929 15:44:25.525764 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.519193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54103 (local address 127.0.97.1:40193)
0929 15:44:25.519477 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.519486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.520066 (+ 580us) server_negotiation.cc:407] Connection header received
0929 15:44:25.520281 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.520288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.520353 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.520489 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.521117 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.521871 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.523800 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.524730 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.524883 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.525003 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.525327 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.525473 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260929 15:44:25.525890 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.518979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42253 (local address 127.0.97.2:35501)
0929 15:44:25.519267 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.519278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.519884 (+ 606us) server_negotiation.cc:407] Connection header received
0929 15:44:25.520222 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.520227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.520297 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.520396 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:25.521010 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.521780 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.523718 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.524723 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.524892 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.525013 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.525469 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.525602 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:44:25.525926 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.518931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40193 (local address 127.0.97.3:54103)
0929 15:44:25.519960 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.519980 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.520103 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.520530 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.520541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.520579 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.520965 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.520973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.522031 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.522041 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.523529 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.523541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.524953 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.524961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.525155 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.525183 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.525302 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.525448 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":473,"threads_started":1}
I20260929 15:44:25.526434 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.518858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.97.3:42253)
0929 15:44:25.519756 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.519773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.519975 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.520467 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.520471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.520501 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:25.520828 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.520840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.521932 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.521937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.523493 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.523510 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.524999 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.525006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.525275 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.525301 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.525433 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.525576 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":31,"thread_start_us":816,"threads_started":1}
I20260929 15:44:25.526726 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1d55a6e19b4be08bf2ab6adcebe498" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594" is_pre_election: true
I20260929 15:44:25.527006 2308 raft_consensus.cc:2454] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 0.
I20260929 15:44:25.527331 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1d55a6e19b4be08bf2ab6adcebe498" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d21183f2464a6593c8c4e0bf0295ce" is_pre_election: true
I20260929 15:44:25.527472 2518 leader_election.cc:302] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f484e8e8a37417581f8a931e34e5594, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:25.527745 2448 raft_consensus.cc:2454] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 0.
I20260929 15:44:25.527899 2664 raft_consensus.cc:2781] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:25.528044 2664 raft_consensus.cc:482] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:25.528134 2664 raft_consensus.cc:3037] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.530418 2664 raft_consensus.cc:504] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.531148 2664 leader_election.cc:288] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 election: Requested vote from peers 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193), 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501)
I20260929 15:44:25.531599 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1d55a6e19b4be08bf2ab6adcebe498" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594"
I20260929 15:44:25.531716 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1d55a6e19b4be08bf2ab6adcebe498" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d21183f2464a6593c8c4e0bf0295ce"
I20260929 15:44:25.531873 2308 raft_consensus.cc:3037] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.531999 2448 raft_consensus.cc:3037] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.534123 2308 raft_consensus.cc:2454] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 1.
I20260929 15:44:25.534329 2448 raft_consensus.cc:2454] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 1.
I20260929 15:44:25.534677 2518 leader_election.cc:302] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f484e8e8a37417581f8a931e34e5594, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:25.535156 2664 raft_consensus.cc:2781] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:25.535645 2664 raft_consensus.cc:686] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [term 1 LEADER]: Becoming Leader. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Running, Role: LEADER
I20260929 15:44:25.536140 2664 consensus_queue.cc:237] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.540323 2681 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: TxnStatusTablet state changed. Reason: New leader b0382bc50fab4883a36c949b5ee7f926. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.541118 2681 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:44:25.542187 2683 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:44:25.542582 2683 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:44:25.542668 2150 catalog_manager.cc:5495] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } health_report { overall_health: HEALTHY } } }
I20260929 15:44:25.573853 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.565949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.0.1:36920)
0929 15:44:25.566249 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.566275 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.566418 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.566983 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.566986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.566996 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:25.567294 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.567310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.568826 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.568835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.570057 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.570226 (+ 169us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.571469 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.571478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.571691 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.571778 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.573279 (+ 1501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.573312 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.573415 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.573594 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 15:44:25.573938 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.565956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.0.97.3:45305)
0929 15:44:25.566191 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.566198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.566366 (+ 168us) server_negotiation.cc:407] Connection header received
0929 15:44:25.566624 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.566630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.566779 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.566958 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:25.567482 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.568670 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.570517 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.571208 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.571384 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.572100 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.573090 (+ 990us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.573301 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.573462 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.573570 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 15:44:25.582909 2308 raft_consensus.cc:1261] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Refusing update from remote peer b0382bc50fab4883a36c949b5ee7f926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:25.582965 2448 raft_consensus.cc:1261] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Refusing update from remote peer b0382bc50fab4883a36c949b5ee7f926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:25.584038 2681 consensus_queue.cc:1035] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:25.584467 2664 consensus_queue.cc:1035] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:25.594135 2665 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: TxnStatusTablet state changed. Reason: New leader b0382bc50fab4883a36c949b5ee7f926. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.594477 2665 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.599529 2663 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: TxnStatusTablet state changed. Reason: New leader b0382bc50fab4883a36c949b5ee7f926. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.599907 2663 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.599875 2686 mvcc.cc:205] Tried to move back new op lower bound from 7334693542217003008 to 7334693542040883200. Current Snapshot: MvccSnapshot[applied={T|T < 7334693542217003008}]
I20260929 15:44:25.602151 2687 mvcc.cc:205] Tried to move back new op lower bound from 7334693542217003008 to 7334693542040883200. Current Snapshot: MvccSnapshot[applied={T|T < 7334693542217003008}]
I20260929 15:44:25.602962 2681 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.603529 2681 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:44:25.603870 2681 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.605409 2681 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P b0382bc50fab4883a36c949b5ee7f926: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:44:25.605681 2688 mvcc.cc:205] Tried to move back new op lower bound from 7334693542217003008 to 7334693542040883200. Current Snapshot: MvccSnapshot[applied={T|T < 7334693542217003008}]
I20260929 15:44:25.615650 2665 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.615952 2665 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 5f484e8e8a37417581f8a931e34e5594: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.620838 2663 tablet_replica.cc:439] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } } }
I20260929 15:44:25.621207 2663 tablet_replica.cc:442] T 4d1d55a6e19b4be08bf2ab6adcebe498 P 12d21183f2464a6593c8c4e0bf0295ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:44:25.631474 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.622406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.0.97.2:35501)
0929 15:44:25.622574 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.622597 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.623197 (+ 600us) server_negotiation.cc:407] Connection header received
0929 15:44:25.623504 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.623509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.623639 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.623803 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:25.624880 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.626269 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.628661 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.629167 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.629316 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.630037 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.630665 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.630806 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.631026 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.631142 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:44:25.654709 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.646928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.97.2:39019)
0929 15:44:25.647651 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.647672 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.647825 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.648412 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.648417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.648431 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:25.648762 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.648772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.650393 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.650399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.652145 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.652161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.653308 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.653314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.653475 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.653559 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.654180 (+ 621us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.654191 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.654275 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.654398 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":413,"threads_started":1}
I20260929 15:44:25.654702 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.647021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39019 (local address 127.0.97.3:45305)
0929 15:44:25.647337 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.647344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.647773 (+ 429us) server_negotiation.cc:407] Connection header received
0929 15:44:25.648078 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.648085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.648261 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.648357 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:25.648917 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.650238 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.652349 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.653086 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.653242 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.653796 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.653952 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.654055 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.654323 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.654406 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260929 15:44:25.669371 2308 raft_consensus.cc:1261] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Refusing update from remote peer 12d21183f2464a6593c8c4e0bf0295ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:25.669713 2588 raft_consensus.cc:1261] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Refusing update from remote peer 12d21183f2464a6593c8c4e0bf0295ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:25.670244 2668 consensus_queue.cc:1035] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:25.671036 2668 consensus_queue.cc:1035] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:25.721801 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.713514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54571 (local address 127.0.97.2:35501)
0929 15:44:25.713739 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.713745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.714420 (+ 675us) server_negotiation.cc:407] Connection header received
0929 15:44:25.714588 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.714595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.714699 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.714810 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:25.715516 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.716698 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.719010 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.719677 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.719825 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.720821 (+ 996us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.721007 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.721084 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.721345 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.721466 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:44:25.721990 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.713511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.97.3:54571)
0929 15:44:25.714302 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.714332 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.714458 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.714859 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.714864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.714876 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:25.715332 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.715345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.716878 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.716883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.718834 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.718853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.720057 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.720066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.720337 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.720512 (+ 175us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.721174 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.721184 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.721301 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.721523 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":407,"threads_started":1}
I20260929 15:44:25.778174 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:25.778487 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:25.786217 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Sending DeleteTablet for 3 replicas of tablet d4ec2d70c4f14943a1f5d5d7a41b676a
I20260929 15:44:25.787860 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ec2d70c4f14943a1f5d5d7a41b676a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60586
I20260929 15:44:25.788420 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ec2d70c4f14943a1f5d5d7a41b676a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50932
I20260929 15:44:25.788857 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ec2d70c4f14943a1f5d5d7a41b676a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36904
I20260929 15:44:25.795367 2709 tablet_replica.cc:330] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: stopping tablet replica
I20260929 15:44:25.796109 2709 raft_consensus.cc:2229] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:25.796660 2709 raft_consensus.cc:2258] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:25.802546 2709 ts_tablet_manager.cc:1905] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:25.792448 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:25.803902 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:25.809674 2707 tablet_replica.cc:330] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: stopping tablet replica
I20260929 15:44:25.810233 2707 raft_consensus.cc:2229] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:25.810655 2707 raft_consensus.cc:2258] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:25.812058 2708 tablet_replica.cc:330] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: stopping tablet replica
I20260929 15:44:25.813735 2288 tablet_service.cc:1467] Processing CreateTablet for tablet 6148d8d6a3504471bd2e3f30ed019218 (DEFAULT_TABLE table=test-table [id=163619bda83a4004b551cdf4b465c2d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:25.814247 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6148d8d6a3504471bd2e3f30ed019218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.813735 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 6148d8d6a3504471bd2e3f30ed019218 (DEFAULT_TABLE table=test-table [id=163619bda83a4004b551cdf4b465c2d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:25.814914 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6148d8d6a3504471bd2e3f30ed019218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.816653 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 6148d8d6a3504471bd2e3f30ed019218 (DEFAULT_TABLE table=test-table [id=163619bda83a4004b551cdf4b465c2d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:25.817150 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6148d8d6a3504471bd2e3f30ed019218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:25.817806 2708 raft_consensus.cc:2229] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:25.821694 2708 raft_consensus.cc:2258] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:25.822062 2707 ts_tablet_manager.cc:1905] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:25.821902 2659 tablet_bootstrap.cc:492] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Bootstrap starting.
I20260929 15:44:25.824600 2659 tablet_bootstrap.cc:654] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.829722 2658 tablet_bootstrap.cc:492] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Bootstrap starting.
I20260929 15:44:25.830971 2657 tablet_bootstrap.cc:492] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Bootstrap starting.
I20260929 15:44:25.832496 2658 tablet_bootstrap.cc:654] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.832847 2657 tablet_bootstrap.cc:654] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:25.842813 2708 ts_tablet_manager.cc:1905] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:25.846027 2709 ts_tablet_manager.cc:1918] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:25.846271 2709 log.cc:1196] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/d4ec2d70c4f14943a1f5d5d7a41b676a
I20260929 15:44:25.846880 2709 ts_tablet_manager.cc:1939] T d4ec2d70c4f14943a1f5d5d7a41b676a P b0382bc50fab4883a36c949b5ee7f926: Deleting consensus metadata
I20260929 15:44:25.849363 2657 tablet_bootstrap.cc:492] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: No bootstrap required, opened a new log
I20260929 15:44:25.849558 2657 ts_tablet_manager.cc:1397] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20260929 15:44:25.850536 2659 tablet_bootstrap.cc:492] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: No bootstrap required, opened a new log
I20260929 15:44:25.850533 2657 raft_consensus.cc:348] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.850862 2657 raft_consensus.cc:374] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.850960 2657 raft_consensus.cc:729] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.851286 2657 consensus_queue.cc:260] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.850915 2659 ts_tablet_manager.cc:1397] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.008s
I20260929 15:44:25.853107 2657 ts_tablet_manager.cc:1428] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 15:44:25.853950 2136 catalog_manager.cc:4841] TS b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305): tablet d4ec2d70c4f14943a1f5d5d7a41b676a (table test-table [id=5fc991670cb44c59b1292c89fb9258cb]) successfully deleted
I20260929 15:44:25.855934 2658 tablet_bootstrap.cc:492] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: No bootstrap required, opened a new log
I20260929 15:44:25.856123 2658 ts_tablet_manager.cc:1397] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260929 15:44:25.856925 2658 raft_consensus.cc:348] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.857174 2658 raft_consensus.cc:374] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.857254 2658 raft_consensus.cc:729] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f484e8e8a37417581f8a931e34e5594, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.857548 2658 consensus_queue.cc:260] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.858222 2658 ts_tablet_manager.cc:1428] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 15:44:25.860467 2708 ts_tablet_manager.cc:1918] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:25.860757 2708 log.cc:1196] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/d4ec2d70c4f14943a1f5d5d7a41b676a
I20260929 15:44:25.861498 2708 ts_tablet_manager.cc:1939] T d4ec2d70c4f14943a1f5d5d7a41b676a P 12d21183f2464a6593c8c4e0bf0295ce: Deleting consensus metadata
I20260929 15:44:25.863602 2136 catalog_manager.cc:4841] TS 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501): tablet d4ec2d70c4f14943a1f5d5d7a41b676a (table test-table [id=5fc991670cb44c59b1292c89fb9258cb]) successfully deleted
I20260929 15:44:25.864058 2659 raft_consensus.cc:348] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.864518 2659 raft_consensus.cc:374] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:25.864701 2659 raft_consensus.cc:729] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Initialized, Role: FOLLOWER
I20260929 15:44:25.865001 2659 consensus_queue.cc:260] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.865921 2659 ts_tablet_manager.cc:1428] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20260929 15:44:25.873802 2707 ts_tablet_manager.cc:1918] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:25.874029 2707 log.cc:1196] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/d4ec2d70c4f14943a1f5d5d7a41b676a
I20260929 15:44:25.874634 2707 ts_tablet_manager.cc:1939] T d4ec2d70c4f14943a1f5d5d7a41b676a P 5f484e8e8a37417581f8a931e34e5594: Deleting consensus metadata
I20260929 15:44:25.880977 2135 catalog_manager.cc:4841] TS 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193): tablet d4ec2d70c4f14943a1f5d5d7a41b676a (table test-table [id=5fc991670cb44c59b1292c89fb9258cb]) successfully deleted
I20260929 15:44:25.891026 2681 raft_consensus.cc:482] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:25.891299 2681 raft_consensus.cc:504] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.892062 2681 leader_election.cc:288] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193), 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501)
I20260929 15:44:25.892824 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6148d8d6a3504471bd2e3f30ed019218" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d21183f2464a6593c8c4e0bf0295ce" is_pre_election: true
I20260929 15:44:25.892869 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6148d8d6a3504471bd2e3f30ed019218" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594" is_pre_election: true
I20260929 15:44:25.893093 2448 raft_consensus.cc:2454] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 0.
I20260929 15:44:25.893098 2308 raft_consensus.cc:2454] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 0.
I20260929 15:44:25.893599 2518 leader_election.cc:302] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f484e8e8a37417581f8a931e34e5594, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:25.894099 2681 raft_consensus.cc:2781] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:25.894248 2681 raft_consensus.cc:482] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:25.894398 2681 raft_consensus.cc:3037] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.897905 2681 raft_consensus.cc:504] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.899472 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6148d8d6a3504471bd2e3f30ed019218" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594"
I20260929 15:44:25.899482 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6148d8d6a3504471bd2e3f30ed019218" candidate_uuid: "b0382bc50fab4883a36c949b5ee7f926" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d21183f2464a6593c8c4e0bf0295ce"
I20260929 15:44:25.899757 2448 raft_consensus.cc:3037] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.899752 2308 raft_consensus.cc:3037] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:25.903774 2308 raft_consensus.cc:2454] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 1.
I20260929 15:44:25.903954 2448 raft_consensus.cc:2454] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0382bc50fab4883a36c949b5ee7f926 in term 1.
I20260929 15:44:25.904680 2681 leader_election.cc:288] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 election: Requested vote from peers 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193), 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501)
I20260929 15:44:25.905167 2519 leader_election.cc:302] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:25.905731 2681 raft_consensus.cc:2781] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:25.905946 2681 raft_consensus.cc:686] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 1 LEADER]: Becoming Leader. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Running, Role: LEADER
I20260929 15:44:25.906904 2681 consensus_queue.cc:237] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:25.918710 2150 catalog_manager.cc:5495] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0382bc50fab4883a36c949b5ee7f926" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } health_report { overall_health: HEALTHY } } }
I20260929 15:44:25.956948 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.949344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.0.97.3:45305)
0929 15:44:25.949557 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.949564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.949763 (+ 199us) server_negotiation.cc:407] Connection header received
0929 15:44:25.950056 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.950061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.950162 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.950269 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:25.950856 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.952104 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.954220 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.955084 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.955283 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.955834 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.956037 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.956214 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.956398 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.956530 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:44:25.977346 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.969933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40943 (local address 127.0.97.3:45305)
0929 15:44:25.970734 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:25.970740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:25.970759 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:44:25.970831 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:25.970838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:25.970933 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:25.971050 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:25.971521 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.972889 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.974992 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.975732 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.975901 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.976519 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.976673 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.976766 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:25.976920 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 15:44:25.977012 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670}
I20260929 15:44:25.977648 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:25.969888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.97.3:40943)
0929 15:44:25.970270 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:25.970294 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:25.970450 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:25.971083 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:25.971089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:25.971103 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:25.971352 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.971363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.973069 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.973075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.974778 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:25.974799 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:25.975947 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:25.975955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:25.976258 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:25.976317 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:25.976774 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:25.976786 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:44:25.976890 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:44:25.977072 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260929 15:44:25.991555 2308 raft_consensus.cc:1261] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Refusing update from remote peer b0382bc50fab4883a36c949b5ee7f926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:25.991650 2448 raft_consensus.cc:1261] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Refusing update from remote peer b0382bc50fab4883a36c949b5ee7f926: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:25.992547 2681 consensus_queue.cc:1035] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:25.993208 2664 consensus_queue.cc:1035] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:44:26.043764 2686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:44:27.676318 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:27.676739 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:27.682156 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Sending DeleteTablet for 3 replicas of tablet 6148d8d6a3504471bd2e3f30ed019218
I20260929 15:44:27.683096 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet 6148d8d6a3504471bd2e3f30ed019218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60586
I20260929 15:44:27.683346 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 6148d8d6a3504471bd2e3f30ed019218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36904
I20260929 15:44:27.683373 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 6148d8d6a3504471bd2e3f30ed019218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50932
I20260929 15:44:27.694988 2737 tablet_replica.cc:330] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: stopping tablet replica
I20260929 15:44:27.695556 2737 raft_consensus.cc:2229] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:27.695982 2737 raft_consensus.cc:2258] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:27.696847 2736 tablet_replica.cc:330] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: stopping tablet replica
I20260929 15:44:27.697170 2736 raft_consensus.cc:2229] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:27.697527 2736 raft_consensus.cc:2258] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:27.698436 2737 ts_tablet_manager.cc:1905] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:27.698849 2736 ts_tablet_manager.cc:1905] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:27.705201 2738 tablet_replica.cc:330] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: stopping tablet replica
I20260929 15:44:27.705439 2738 raft_consensus.cc:2229] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:27.706049 2738 raft_consensus.cc:2258] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:27.706972 2738 ts_tablet_manager.cc:1905] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:27.707674 2736 ts_tablet_manager.cc:1918] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:44:27.707857 2736 log.cc:1196] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/6148d8d6a3504471bd2e3f30ed019218
I20260929 15:44:27.708387 2736 ts_tablet_manager.cc:1939] T 6148d8d6a3504471bd2e3f30ed019218 P 5f484e8e8a37417581f8a931e34e5594: Deleting consensus metadata
I20260929 15:44:27.709262 2737 ts_tablet_manager.cc:1918] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:44:27.709419 2737 log.cc:1196] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/6148d8d6a3504471bd2e3f30ed019218
I20260929 15:44:27.709712 2135 catalog_manager.cc:4841] TS 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193): tablet 6148d8d6a3504471bd2e3f30ed019218 (table test-table [id=163619bda83a4004b551cdf4b465c2d4]) successfully deleted
I20260929 15:44:27.709895 2737 ts_tablet_manager.cc:1939] T 6148d8d6a3504471bd2e3f30ed019218 P 12d21183f2464a6593c8c4e0bf0295ce: Deleting consensus metadata
I20260929 15:44:27.711112 2136 catalog_manager.cc:4841] TS 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501): tablet 6148d8d6a3504471bd2e3f30ed019218 (table test-table [id=163619bda83a4004b551cdf4b465c2d4]) successfully deleted
I20260929 15:44:27.713589 2738 ts_tablet_manager.cc:1918] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:44:27.713714 2738 log.cc:1196] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/6148d8d6a3504471bd2e3f30ed019218
I20260929 15:44:27.714177 2738 ts_tablet_manager.cc:1939] T 6148d8d6a3504471bd2e3f30ed019218 P b0382bc50fab4883a36c949b5ee7f926: Deleting consensus metadata
I20260929 15:44:27.715320 2136 catalog_manager.cc:4841] TS b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305): tablet 6148d8d6a3504471bd2e3f30ed019218 (table test-table [id=163619bda83a4004b551cdf4b465c2d4]) successfully deleted
Sep 29 15:44:27 dist-test-slave-7491 krb5kdc[2106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:44:27 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:27.758315 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.733959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.0.97.62:44185)
0929 15:44:27.734780 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:27.734786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:27.734819 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:44:27.734901 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:27.734905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:27.734983 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:27.735130 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:27.736638 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.737527 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.739161 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.739604 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.740690 (+ 1086us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.751552 (+ 10862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:27.751559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:27.751563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:27.751608 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:27.754838 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:27.755828 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:27.755833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:27.755836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:27.756026 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:27.756628 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:27.756633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:27.756636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:27.757004 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:27.757257 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:27.757658 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 15:44:27.757893 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":471,"threads_started":1}
I20260929 15:44:27.763195 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:27.763952 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:27.773381 2288 tablet_service.cc:1467] Processing CreateTablet for tablet cc9eb47a22b54f3695d8c9341b8f7dfc (DEFAULT_TABLE table=test-table [id=8bd2d5c945ec448f8e82fb03bb6abf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:27.773940 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9eb47a22b54f3695d8c9341b8f7dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:27.773965 2428 tablet_service.cc:1467] Processing CreateTablet for tablet cc9eb47a22b54f3695d8c9341b8f7dfc (DEFAULT_TABLE table=test-table [id=8bd2d5c945ec448f8e82fb03bb6abf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:27.774091 2568 tablet_service.cc:1467] Processing CreateTablet for tablet cc9eb47a22b54f3695d8c9341b8f7dfc (DEFAULT_TABLE table=test-table [id=8bd2d5c945ec448f8e82fb03bb6abf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:27.774362 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9eb47a22b54f3695d8c9341b8f7dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:27.774803 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9eb47a22b54f3695d8c9341b8f7dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:27.784545 2747 tablet_bootstrap.cc:492] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Bootstrap starting.
I20260929 15:44:27.786036 2747 tablet_bootstrap.cc:654] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:27.787619 2749 tablet_bootstrap.cc:492] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Bootstrap starting.
I20260929 15:44:27.788738 2748 tablet_bootstrap.cc:492] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Bootstrap starting.
I20260929 15:44:27.788898 2747 tablet_bootstrap.cc:492] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: No bootstrap required, opened a new log
I20260929 15:44:27.789053 2747 ts_tablet_manager.cc:1397] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 15:44:27.789676 2749 tablet_bootstrap.cc:654] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:27.789964 2747 raft_consensus.cc:348] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.790230 2747 raft_consensus.cc:374] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:27.790330 2747 raft_consensus.cc:729] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Initialized, Role: FOLLOWER
I20260929 15:44:27.791531 2747 consensus_queue.cc:260] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.792651 2748 tablet_bootstrap.cc:654] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:27.794526 2747 ts_tablet_manager.cc:1428] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:44:27.798233 2748 tablet_bootstrap.cc:492] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: No bootstrap required, opened a new log
I20260929 15:44:27.798455 2748 ts_tablet_manager.cc:1397] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 15:44:27.799439 2748 raft_consensus.cc:348] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.799705 2749 tablet_bootstrap.cc:492] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: No bootstrap required, opened a new log
I20260929 15:44:27.799750 2748 raft_consensus.cc:374] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:27.799876 2748 raft_consensus.cc:729] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f484e8e8a37417581f8a931e34e5594, State: Initialized, Role: FOLLOWER
I20260929 15:44:27.800069 2749 ts_tablet_manager.cc:1397] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260929 15:44:27.800159 2748 consensus_queue.cc:260] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.801017 2749 raft_consensus.cc:348] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.801271 2749 raft_consensus.cc:374] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:27.801357 2749 raft_consensus.cc:729] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Initialized, Role: FOLLOWER
I20260929 15:44:27.801604 2749 consensus_queue.cc:260] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.802629 2749 ts_tablet_manager.cc:1428] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:44:27.803481 2748 ts_tablet_manager.cc:1428] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 15:44:27.890221 2753 raft_consensus.cc:482] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:27.890513 2753 raft_consensus.cc:504] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.892887 2753 leader_election.cc:288] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501), b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305)
I20260929 15:44:27.901810 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.893940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.97.1:49775)
0929 15:44:27.894878 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:27.894899 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:27.895024 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:27.895401 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:27.895410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:27.895435 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:27.895773 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:27.895781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.896930 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.896935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:27.899084 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:27.899102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.900633 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.900642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:27.900863 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.900899 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:44:27.901052 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:44:27.901242 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":599,"threads_started":1}
I20260929 15:44:27.901808 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.893224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58879 (local address 127.0.97.2:35501)
0929 15:44:27.894622 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:27.894631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:27.894661 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:44:27.894755 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:27.894763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:27.894869 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:27.895016 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:27.895546 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.896381 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.899247 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.900515 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.900738 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.900894 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:27.901283 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 15:44:27.901438 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":1054,"threads_started":1}
I20260929 15:44:27.901798 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.894386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49775 (local address 127.0.97.3:45305)
0929 15:44:27.895115 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:27.895123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:27.895145 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:44:27.895213 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:27.895216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:27.895268 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:27.895386 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:27.895934 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.896790 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.899371 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.900370 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.900547 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.900722 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:27.901098 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 15:44:27.901262 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":458,"threads_started":1}
I20260929 15:44:27.901899 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.893137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35501 (local address 127.0.97.1:58879)
0929 15:44:27.893808 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:27.893834 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:27.893972 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:27.895042 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:27.895047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:27.895075 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:27.895375 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:27.895383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.896544 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.896554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:27.898917 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:27.898947 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.900869 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.900878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:27.901090 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.901122 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:44:27.901251 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 15:44:27.901426 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":5,"thread_start_us":323,"threads_started":1}
I20260929 15:44:27.902999 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9eb47a22b54f3695d8c9341b8f7dfc" candidate_uuid: "5f484e8e8a37417581f8a931e34e5594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0382bc50fab4883a36c949b5ee7f926" is_pre_election: true
I20260929 15:44:27.903025 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9eb47a22b54f3695d8c9341b8f7dfc" candidate_uuid: "5f484e8e8a37417581f8a931e34e5594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d21183f2464a6593c8c4e0bf0295ce" is_pre_election: true
I20260929 15:44:27.903287 2447 raft_consensus.cc:2454] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f484e8e8a37417581f8a931e34e5594 in term 0.
I20260929 15:44:27.903272 2587 raft_consensus.cc:2454] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f484e8e8a37417581f8a931e34e5594 in term 0.
I20260929 15:44:27.903848 2239 leader_election.cc:302] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, 5f484e8e8a37417581f8a931e34e5594; no voters:
I20260929 15:44:27.904394 2753 raft_consensus.cc:2781] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:27.904561 2753 raft_consensus.cc:482] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:27.904693 2753 raft_consensus.cc:3037] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:27.906899 2753 raft_consensus.cc:504] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.907538 2753 leader_election.cc:288] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [CANDIDATE]: Term 1 election: Requested vote from peers 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501), b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305)
I20260929 15:44:27.908159 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9eb47a22b54f3695d8c9341b8f7dfc" candidate_uuid: "5f484e8e8a37417581f8a931e34e5594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d21183f2464a6593c8c4e0bf0295ce"
I20260929 15:44:27.908298 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9eb47a22b54f3695d8c9341b8f7dfc" candidate_uuid: "5f484e8e8a37417581f8a931e34e5594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0382bc50fab4883a36c949b5ee7f926"
I20260929 15:44:27.908442 2447 raft_consensus.cc:3037] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:27.908530 2587 raft_consensus.cc:3037] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:27.910647 2447 raft_consensus.cc:2454] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f484e8e8a37417581f8a931e34e5594 in term 1.
I20260929 15:44:27.911221 2587 raft_consensus.cc:2454] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f484e8e8a37417581f8a931e34e5594 in term 1.
I20260929 15:44:27.911234 2239 leader_election.cc:302] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, 5f484e8e8a37417581f8a931e34e5594; no voters:
I20260929 15:44:27.911604 2753 raft_consensus.cc:2781] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:27.912227 2753 raft_consensus.cc:686] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 1 LEADER]: Becoming Leader. State: Replica: 5f484e8e8a37417581f8a931e34e5594, State: Running, Role: LEADER
I20260929 15:44:27.912683 2753 consensus_queue.cc:237] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:27.917137 2151 catalog_manager.cc:5495] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5f484e8e8a37417581f8a931e34e5594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } health_report { overall_health: UNKNOWN } } }
I20260929 15:44:27.971134 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.964769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.0.97.1:40193)
0929 15:44:27.965497 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:27.965503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:27.965531 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:44:27.965625 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:27.965629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:27.965727 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:27.965908 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:27.966337 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.967174 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.968618 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.969039 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.969173 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.969659 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:27.970338 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:27.970508 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:27.970628 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 15:44:27.970730 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":388,"threads_started":1}
I20260929 15:44:27.992997 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.987296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60269 (local address 127.0.97.3:45305)
0929 15:44:27.987605 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:27.987611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:27.988013 (+ 402us) server_negotiation.cc:407] Connection header received
0929 15:44:27.988277 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:27.988281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:27.988351 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:27.988455 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:27.988971 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.989717 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.991134 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.991561 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.991710 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.992151 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:27.992358 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:27.992453 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:27.992600 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 15:44:27.992683 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 15:44:27.993112 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:27.987035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45305 (local address 127.0.97.1:60269)
0929 15:44:27.987896 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:27.987912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:27.988106 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:27.988482 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:27.988487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:27.988498 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:27.988813 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:27.988822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.989885 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.989891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:27.990971 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:27.990983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:27.991751 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:27.991759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:27.991890 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:27.991955 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:27.992454 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:27.992467 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 15:44:27.992579 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 15:44:27.992715 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":519,"threads_started":1}
I20260929 15:44:28.003365 2447 raft_consensus.cc:1261] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Refusing update from remote peer 5f484e8e8a37417581f8a931e34e5594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:28.003324 2587 raft_consensus.cc:1261] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Refusing update from remote peer 5f484e8e8a37417581f8a931e34e5594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:28.004227 2759 consensus_queue.cc:1035] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:28.004638 2753 consensus_queue.cc:1035] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:28.040776 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:28.033312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51869 (local address 127.0.97.1:40193)
0929 15:44:28.033527 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:28.033536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:28.034048 (+ 512us) server_negotiation.cc:407] Connection header received
0929 15:44:28.034280 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:28.034286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:28.034371 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:28.034469 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:28.034967 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:28.035956 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:28.038555 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:28.039129 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:28.039269 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:28.039871 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:28.040011 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:28.040096 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:28.040386 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 15:44:28.040462 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:44:28.040776 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:28.033168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40193 (local address 127.0.97.3:51869)
0929 15:44:28.033918 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:28.033936 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:28.034128 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:28.034497 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:28.034501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:28.034513 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:44:28.034796 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:28.034805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:28.036225 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:28.036230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:28.038273 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:28.038296 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:28.039420 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:28.039426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:28.039595 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:28.039643 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:44:28.040214 (+ 571us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:44:28.040225 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:44:28.040309 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:44:28.040452 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":467,"threads_started":1}
I20260929 15:44:28.083307 2150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:28.083546 2150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:28.086892 2150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Sending DeleteTablet for 3 replicas of tablet cc9eb47a22b54f3695d8c9341b8f7dfc
I20260929 15:44:28.088078 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9eb47a22b54f3695d8c9341b8f7dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50932
I20260929 15:44:28.088086 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9eb47a22b54f3695d8c9341b8f7dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36904
I20260929 15:44:28.088429 2737 tablet_replica.cc:330] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: stopping tablet replica
I20260929 15:44:28.088439 2738 tablet_replica.cc:330] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: stopping tablet replica
I20260929 15:44:28.088641 2737 raft_consensus.cc:2229] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:28.088662 2738 raft_consensus.cc:2229] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:28.088846 2737 raft_consensus.cc:2258] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:28.088852 2738 raft_consensus.cc:2258] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:28.090157 2737 ts_tablet_manager.cc:1905] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:28.090164 2738 ts_tablet_manager.cc:1905] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:28.096762 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9eb47a22b54f3695d8c9341b8f7dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60586
I20260929 15:44:28.097095 2736 tablet_replica.cc:330] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: stopping tablet replica
I20260929 15:44:28.097307 2736 raft_consensus.cc:2229] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:28.097894 2736 raft_consensus.cc:2258] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:28.098006 2738 ts_tablet_manager.cc:1918] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:28.098296 2738 log.cc:1196] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/cc9eb47a22b54f3695d8c9341b8f7dfc
I20260929 15:44:28.099396 2738 ts_tablet_manager.cc:1939] T cc9eb47a22b54f3695d8c9341b8f7dfc P b0382bc50fab4883a36c949b5ee7f926: Deleting consensus metadata
I20260929 15:44:28.100453 2736 ts_tablet_manager.cc:1905] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:28.101387 2136 catalog_manager.cc:4841] TS b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305): tablet cc9eb47a22b54f3695d8c9341b8f7dfc (table test-table [id=8bd2d5c945ec448f8e82fb03bb6abf4a]) successfully deleted
I20260929 15:44:28.103744 2737 ts_tablet_manager.cc:1918] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:28.103929 2737 log.cc:1196] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/cc9eb47a22b54f3695d8c9341b8f7dfc
I20260929 15:44:28.104449 2737 ts_tablet_manager.cc:1939] T cc9eb47a22b54f3695d8c9341b8f7dfc P 12d21183f2464a6593c8c4e0bf0295ce: Deleting consensus metadata
I20260929 15:44:28.105855 2136 catalog_manager.cc:4841] TS 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501): tablet cc9eb47a22b54f3695d8c9341b8f7dfc (table test-table [id=8bd2d5c945ec448f8e82fb03bb6abf4a]) successfully deleted
I20260929 15:44:28.105947 2150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:28.106726 2150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:28.107167 2736 ts_tablet_manager.cc:1918] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:44:28.107363 2736 log.cc:1196] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/cc9eb47a22b54f3695d8c9341b8f7dfc
I20260929 15:44:28.107772 2736 ts_tablet_manager.cc:1939] T cc9eb47a22b54f3695d8c9341b8f7dfc P 5f484e8e8a37417581f8a931e34e5594: Deleting consensus metadata
I20260929 15:44:28.110013 2135 catalog_manager.cc:4841] TS 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193): tablet cc9eb47a22b54f3695d8c9341b8f7dfc (table test-table [id=8bd2d5c945ec448f8e82fb03bb6abf4a]) successfully deleted
I20260929 15:44:28.114955 2428 tablet_service.cc:1467] Processing CreateTablet for tablet 7115e15c30b24401a53bfbfca29604a5 (DEFAULT_TABLE table=test-table [id=198f57ca882e4a12ae6efc820ada4dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:28.114959 2288 tablet_service.cc:1467] Processing CreateTablet for tablet 7115e15c30b24401a53bfbfca29604a5 (DEFAULT_TABLE table=test-table [id=198f57ca882e4a12ae6efc820ada4dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:28.115469 2288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7115e15c30b24401a53bfbfca29604a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:28.115468 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7115e15c30b24401a53bfbfca29604a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:28.123286 2748 tablet_bootstrap.cc:492] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Bootstrap starting.
I20260929 15:44:28.124431 2749 tablet_bootstrap.cc:492] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Bootstrap starting.
I20260929 15:44:28.126655 2749 tablet_bootstrap.cc:654] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:28.128013 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 7115e15c30b24401a53bfbfca29604a5 (DEFAULT_TABLE table=test-table [id=198f57ca882e4a12ae6efc820ada4dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:28.128543 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7115e15c30b24401a53bfbfca29604a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:28.132334 2748 tablet_bootstrap.cc:654] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:28.137861 2747 tablet_bootstrap.cc:492] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Bootstrap starting.
I20260929 15:44:28.138516 2749 tablet_bootstrap.cc:492] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: No bootstrap required, opened a new log
I20260929 15:44:28.138722 2749 ts_tablet_manager.cc:1397] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20260929 15:44:28.139868 2748 tablet_bootstrap.cc:492] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: No bootstrap required, opened a new log
I20260929 15:44:28.139882 2747 tablet_bootstrap.cc:654] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:28.139824 2749 raft_consensus.cc:348] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.140193 2749 raft_consensus.cc:374] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:28.140298 2749 raft_consensus.cc:729] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Initialized, Role: FOLLOWER
I20260929 15:44:28.140357 2748 ts_tablet_manager.cc:1397] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260929 15:44:28.140659 2749 consensus_queue.cc:260] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.141426 2748 raft_consensus.cc:348] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.141769 2748 raft_consensus.cc:374] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:28.141901 2748 raft_consensus.cc:729] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f484e8e8a37417581f8a931e34e5594, State: Initialized, Role: FOLLOWER
I20260929 15:44:28.142295 2748 consensus_queue.cc:260] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.143674 2748 ts_tablet_manager.cc:1428] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:44:28.144590 2749 ts_tablet_manager.cc:1428] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260929 15:44:28.147395 2747 tablet_bootstrap.cc:492] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: No bootstrap required, opened a new log
I20260929 15:44:28.147590 2747 ts_tablet_manager.cc:1397] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260929 15:44:28.148370 2747 raft_consensus.cc:348] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.148622 2747 raft_consensus.cc:374] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:28.148701 2747 raft_consensus.cc:729] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0382bc50fab4883a36c949b5ee7f926, State: Initialized, Role: FOLLOWER
I20260929 15:44:28.148947 2747 consensus_queue.cc:260] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.149514 2747 ts_tablet_manager.cc:1428] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:44:28.222180 2754 raft_consensus.cc:482] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:28.222462 2754 raft_consensus.cc:504] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.223264 2754 leader_election.cc:288] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193), b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305)
I20260929 15:44:28.223992 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7115e15c30b24401a53bfbfca29604a5" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0382bc50fab4883a36c949b5ee7f926" is_pre_election: true
I20260929 15:44:28.224298 2587 raft_consensus.cc:2454] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 0.
I20260929 15:44:28.224225 2307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7115e15c30b24401a53bfbfca29604a5" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594" is_pre_election: true
I20260929 15:44:28.224570 2307 raft_consensus.cc:2454] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 0.
I20260929 15:44:28.224774 2379 leader_election.cc:302] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:28.225069 2754 raft_consensus.cc:2781] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:28.225205 2754 raft_consensus.cc:482] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:28.225301 2754 raft_consensus.cc:3037] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:28.253726 2754 raft_consensus.cc:504] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.254863 2754 leader_election.cc:288] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 election: Requested vote from peers 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193), b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305)
I20260929 15:44:28.255376 2307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7115e15c30b24401a53bfbfca29604a5" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f484e8e8a37417581f8a931e34e5594"
I20260929 15:44:28.255638 2307 raft_consensus.cc:3037] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:28.255923 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7115e15c30b24401a53bfbfca29604a5" candidate_uuid: "12d21183f2464a6593c8c4e0bf0295ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0382bc50fab4883a36c949b5ee7f926"
I20260929 15:44:28.256150 2587 raft_consensus.cc:3037] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:28.341084 2587 raft_consensus.cc:2454] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 1.
I20260929 15:44:28.341187 2307 raft_consensus.cc:2454] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d21183f2464a6593c8c4e0bf0295ce in term 1.
I20260929 15:44:28.341821 2379 leader_election.cc:302] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d21183f2464a6593c8c4e0bf0295ce, b0382bc50fab4883a36c949b5ee7f926; no voters:
I20260929 15:44:28.342226 2754 raft_consensus.cc:2781] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:28.342890 2754 raft_consensus.cc:686] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 LEADER]: Becoming Leader. State: Replica: 12d21183f2464a6593c8c4e0bf0295ce, State: Running, Role: LEADER
I20260929 15:44:28.343626 2754 consensus_queue.cc:237] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } }
I20260929 15:44:28.346967 2150 catalog_manager.cc:5495] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2). New cstate: current_term: 1 leader_uuid: "12d21183f2464a6593c8c4e0bf0295ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d21183f2464a6593c8c4e0bf0295ce" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 } health_report { overall_health: UNKNOWN } } }
I20260929 15:44:28.399621 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:28.393586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.0.97.2:35501)
0929 15:44:28.393814 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:28.393821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:28.393842 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:44:28.394096 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:28.394106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:28.394174 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:28.394283 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:28.394723 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:28.395480 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:28.397121 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:28.397526 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:28.397679 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:28.398457 (+ 778us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:28.398603 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:28.398769 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:28.398942 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 15:44:28.399028 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:44:28.412947 2587 raft_consensus.cc:1261] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Refusing update from remote peer 12d21183f2464a6593c8c4e0bf0295ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:28.413195 2307 raft_consensus.cc:1261] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Refusing update from remote peer 12d21183f2464a6593c8c4e0bf0295ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:28.413656 2776 consensus_queue.cc:1035] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0382bc50fab4883a36c949b5ee7f926" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:28.414121 2754 consensus_queue.cc:1035] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f484e8e8a37417581f8a931e34e5594" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:44:28.446002 2720 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:44:30.064881 2149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:30.065171 2149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:30.070808 2149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7c1c66915e45f4a27c4453d0d7a41b: Sending DeleteTablet for 3 replicas of tablet 7115e15c30b24401a53bfbfca29604a5
I20260929 15:44:30.071771 2288 tablet_service.cc:1514] Processing DeleteTablet for tablet 7115e15c30b24401a53bfbfca29604a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60586
I20260929 15:44:30.072342 2428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7115e15c30b24401a53bfbfca29604a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50932
I20260929 15:44:30.072826 2567 tablet_service.cc:1514] Processing DeleteTablet for tablet 7115e15c30b24401a53bfbfca29604a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36904
I20260929 15:44:30.074559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2222
I20260929 15:44:30.074853 2803 tablet_replica.cc:330] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: stopping tablet replica
I20260929 15:44:30.075135 2803 raft_consensus.cc:2229] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:30.075489 2803 raft_consensus.cc:2258] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:30.076880 2803 ts_tablet_manager.cc:1905] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:30.084205 2803 ts_tablet_manager.cc:1918] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:44:30.084461 2803 log.cc:1196] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/7115e15c30b24401a53bfbfca29604a5
I20260929 15:44:30.084486 2805 tablet_replica.cc:330] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: stopping tablet replica
I20260929 15:44:30.084904 2805 raft_consensus.cc:2229] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:30.085022 2803 ts_tablet_manager.cc:1939] T 7115e15c30b24401a53bfbfca29604a5 P 5f484e8e8a37417581f8a931e34e5594: Deleting consensus metadata
I20260929 15:44:30.085368 2805 raft_consensus.cc:2258] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:30.086136 2804 tablet_replica.cc:330] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: stopping tablet replica
I20260929 15:44:30.086436 2804 raft_consensus.cc:2229] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:30.086527 2135 catalog_manager.cc:4841] TS 5f484e8e8a37417581f8a931e34e5594 (127.0.97.1:40193): tablet 7115e15c30b24401a53bfbfca29604a5 (table test-table [id=198f57ca882e4a12ae6efc820ada4dc8]) successfully deleted
I20260929 15:44:30.087028 2804 raft_consensus.cc:2258] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:30.087198 2805 ts_tablet_manager.cc:1905] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:30.088291 2804 ts_tablet_manager.cc:1905] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:30.093452 2805 ts_tablet_manager.cc:1918] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:44:30.093732 2805 log.cc:1196] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/7115e15c30b24401a53bfbfca29604a5
I20260929 15:44:30.094236 2805 ts_tablet_manager.cc:1939] T 7115e15c30b24401a53bfbfca29604a5 P b0382bc50fab4883a36c949b5ee7f926: Deleting consensus metadata
Sep 29 15:44:30 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:30.096002 2136 catalog_manager.cc:4841] TS b0382bc50fab4883a36c949b5ee7f926 (127.0.97.3:45305): tablet 7115e15c30b24401a53bfbfca29604a5 (table test-table [id=198f57ca882e4a12ae6efc820ada4dc8]) successfully deleted
I20260929 15:44:30.096680 2804 ts_tablet_manager.cc:1918] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:44:30.096853 2804 log.cc:1196] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/7115e15c30b24401a53bfbfca29604a5
I20260929 15:44:30.097393 2804 ts_tablet_manager.cc:1939] T 7115e15c30b24401a53bfbfca29604a5 P 12d21183f2464a6593c8c4e0bf0295ce: Deleting consensus metadata
I20260929 15:44:30.099201 2136 catalog_manager.cc:4841] TS 12d21183f2464a6593c8c4e0bf0295ce (127.0.97.2:35501): tablet 7115e15c30b24401a53bfbfca29604a5 (table test-table [id=198f57ca882e4a12ae6efc820ada4dc8]) successfully deleted
I20260929 15:44:30.105031 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:30.075619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.97.1:40193)
0929 15:44:30.081502 (+ 5883us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:30.081509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:30.081542 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:44:30.081620 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:30.081626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:30.081726 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:30.081862 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:30.083845 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:30.085587 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:30.089017 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:30.089794 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:30.089954 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:30.097218 (+ 7264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:30.097245 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:30.097272 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:30.097333 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:30.101295 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:30.102361 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:30.102370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:30.102372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:30.102560 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:30.103196 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:30.103199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:30.103201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:30.103600 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:30.103924 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:30.104447 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 15:44:30.104681 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5716,"thread_start_us":5606,"threads_started":1}
W20260929 15:44:30.105818 2353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600
W20260929 15:44:30.106573 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2222: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:30.106753 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 2222
I20260929 15:44:30.137382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2362
Sep 29 15:44:30 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:30.157378 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:30.138459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.0.97.2:35501)
0929 15:44:30.139199 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:30.139203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:30.139224 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:44:30.139288 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:30.139291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:30.139356 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:30.139495 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:30.141384 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:30.142254 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:30.143888 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:30.144512 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:30.144652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:30.149796 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:30.149832 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:30.149849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:30.149896 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:30.154108 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:30.155063 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:30.155072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:30.155075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:30.155224 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:30.155633 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:30.155639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:30.155642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:30.156032 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:30.156414 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:30.156747 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 15:44:30.156997 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":360,"threads_started":1}
W20260929 15:44:30.158087 2493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50964
W20260929 15:44:30.159029 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:30.159219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 2362
I20260929 15:44:30.187253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2502
Sep 29 15:44:30 dist-test-slave-7491 krb5kdc[2106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:30.207072 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:30.188255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.0.97.3:45305)
0929 15:44:30.188848 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:30.188852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:30.188870 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:44:30.188927 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:30.188931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:30.188998 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:30.189097 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:30.190560 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:30.191376 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:30.192948 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:30.193562 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:30.193734 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:30.199377 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:30.199401 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:30.199425 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:30.199494 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:30.203103 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:30.204260 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:30.204275 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:30.204282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:30.204486 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:30.204981 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:30.204986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:30.204988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:30.205683 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:30.206077 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:30.206553 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 15:44:30.206775 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":323,"threads_started":1}
W20260929 15:44:30.207828 2633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36948
W20260929 15:44:30.208606 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:30.208786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 2502
I20260929 15:44:30.239058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2119
I20260929 15:44:30.240453 2191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:30.375799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 2119
2026-09-29T15:44:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9658 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:30 dist-test-slave-7491 krb5kdc[2813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:30 dist-test-slave-7491 krb5kdc[2813](info): set up 2 sockets
Sep 29 15:44:30 dist-test-slave-7491 krb5kdc[2813](info): commencing operation
krb5kdc: starting...
W20260929 15:44:31.517207 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:31 dist-test-slave-7491 krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:31.741338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41969
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:32.129485 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:32.129956 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:32.130136 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:32.138019 2826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:32.138216 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:32.138314 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:32.138419 2826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:32.138499 2826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:32.149389 2826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:32.150964 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:32.152513 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:32.166213 2834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:32.166213 2832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:32.166466 2831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:32.167076 2826 server_base.cc:1034] running on GCE node
I20260929 15:44:32.168076 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:32.169996 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:32.171307 2826 hybrid_clock.cc:648] HybridClock initialized: now 1790696672171263 us; error 71 us; skew 500 ppm
I20260929 15:44:32.171721 2826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:32 dist-test-slave-7491 krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:32.175727 2826 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:32.178200 2826 webserver.cc:460] Webserver started at http://127.0.97.62:46693/ using document root <none> and password file <none>
I20260929 15:44:32.179446 2826 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:32.179700 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:32.180104 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:32.183730 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "5a37e750a22b4c189f7332b99a8b6faf"
format_stamp: "Formatted at 2026-09-29 15:44:32 on dist-test-slave-7491"
server_key: "69b65e116eebbc9d29595b87b4f61738"
server_key_iv: "b8452223391ecbb165985a848afb153a"
server_key_version: "encryptionkey@0"
I20260929 15:44:32.185204 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a37e750a22b4c189f7332b99a8b6faf"
format_stamp: "Formatted at 2026-09-29 15:44:32 on dist-test-slave-7491"
server_key: "69b65e116eebbc9d29595b87b4f61738"
server_key_iv: "b8452223391ecbb165985a848afb153a"
server_key_version: "encryptionkey@0"
I20260929 15:44:32.193468 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 15:44:32.198127 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:32.200378 2826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:44:32.200666 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "5a37e750a22b4c189f7332b99a8b6faf"
format_stamp: "Formatted at 2026-09-29 15:44:32 on dist-test-slave-7491"
server_key: "69b65e116eebbc9d29595b87b4f61738"
server_key_iv: "b8452223391ecbb165985a848afb153a"
server_key_version: "encryptionkey@0"
I20260929 15:44:32.200927 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:32.221724 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:32.228634 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:32.229043 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:32.252911 2826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41969
I20260929 15:44:32.252960 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41969 every 8 connection(s)
I20260929 15:44:32.255206 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:32.255961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 2826
I20260929 15:44:32.256335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:32.257144 388 external_mini_cluster.cc:1428] Setting key 439c743b44c196b7037371ad9edc3d12
I20260929 15:44:32.260607 2893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:44:32 dist-test-slave-7491 krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:32.276594 2893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf: Bootstrap starting.
I20260929 15:44:32.280723 2893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:32.281951 2893 log.cc:824] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:32.285385 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:32.260233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.97.62:41969)
0929 15:44:32.261185 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:32.261198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:32.261245 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:44:32.263005 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:32.263032 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:32.263506 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:32.263901 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:32.266132 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:32.267610 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:32.269584 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:32.270383 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:32.270549 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:32.276026 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:32.276036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:32.276055 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:32.276120 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:32.279991 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:32.281161 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:32.281170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:32.281183 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:32.281411 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:32.281864 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:32.281869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:32.281871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:32.282652 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:32.283071 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:32.283687 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 15:44:32.284307 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":433,"threads_started":1}
I20260929 15:44:32.285593 2893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf: No bootstrap required, opened a new log
I20260929 15:44:32.295011 2893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37e750a22b4c189f7332b99a8b6faf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41969 } }
I20260929 15:44:32.295527 2893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:32.295657 2893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a37e750a22b4c189f7332b99a8b6faf, State: Initialized, Role: FOLLOWER
I20260929 15:44:32.296402 2893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37e750a22b4c189f7332b99a8b6faf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41969 } }
I20260929 15:44:32.296693 2893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:32.296851 2893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:32.297040 2893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:32.299774 2893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37e750a22b4c189f7332b99a8b6faf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41969 } }
I20260929 15:44:32.300508 2893 leader_election.cc:302] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a37e750a22b4c189f7332b99a8b6faf; no voters:
I20260929 15:44:32.301226 2893 leader_election.cc:288] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:32.301416 2898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:32.302143 2898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [term 1 LEADER]: Becoming Leader. State: Replica: 5a37e750a22b4c189f7332b99a8b6faf, State: Running, Role: LEADER
I20260929 15:44:32.302774 2898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37e750a22b4c189f7332b99a8b6faf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41969 } }
I20260929 15:44:32.303773 2893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:32.306774 2899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a37e750a22b4c189f7332b99a8b6faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37e750a22b4c189f7332b99a8b6faf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41969 } } }
I20260929 15:44:32.306767 2900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a37e750a22b4c189f7332b99a8b6faf. Latest consensus state: current_term: 1 leader_uuid: "5a37e750a22b4c189f7332b99a8b6faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37e750a22b4c189f7332b99a8b6faf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41969 } } }
I20260929 15:44:32.308194 2899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:32.308287 2900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:32.312489 2905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:32.318136 2905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:32.329516 2905 catalog_manager.cc:1347] Generated new cluster ID: b8335fc6db5d47c7b286c795ab01311e
I20260929 15:44:32.329674 2905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:32.344486 2905 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:32.347097 2905 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:32.384979 2905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a37e750a22b4c189f7332b99a8b6faf: Generated new TSK 0
I20260929 15:44:32.386016 2905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:32.448302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41969
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:32.859126 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:32.859457 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:32.859551 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:32.867395 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:32.867543 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:32.867702 2919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:32.878198 2919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:32.879676 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:32.881346 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:32.896303 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:32.896261 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:32.896550 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:32.897703 2919 server_base.cc:1034] running on GCE node
I20260929 15:44:32.898408 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:32.899462 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:32.900756 2919 hybrid_clock.cc:648] HybridClock initialized: now 1790696672900701 us; error 49 us; skew 500 ppm
I20260929 15:44:32.901182 2919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:32 dist-test-slave-7491 krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:32.914012 2919 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:32.915784 2919 webserver.cc:460] Webserver started at http://127.0.97.1:38331/ using document root <none> and password file <none>
I20260929 15:44:32.916615 2919 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:32.916744 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:32.917013 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:32.920701 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "c26d05b6cad246b5bc01eff5838ccd51"
format_stamp: "Formatted at 2026-09-29 15:44:32 on dist-test-slave-7491"
server_key: "8b9f00ed97bf65bb4081018dc00b86de"
server_key_iv: "158c38f657e01f347bd9c9e4d01c4b7f"
server_key_version: "encryptionkey@0"
I20260929 15:44:32.922194 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c26d05b6cad246b5bc01eff5838ccd51"
format_stamp: "Formatted at 2026-09-29 15:44:32 on dist-test-slave-7491"
server_key: "8b9f00ed97bf65bb4081018dc00b86de"
server_key_iv: "158c38f657e01f347bd9c9e4d01c4b7f"
server_key_version: "encryptionkey@0"
I20260929 15:44:32.928150 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:44:32.932407 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:32.934010 2919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:44:32.934234 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "c26d05b6cad246b5bc01eff5838ccd51"
format_stamp: "Formatted at 2026-09-29 15:44:32 on dist-test-slave-7491"
server_key: "8b9f00ed97bf65bb4081018dc00b86de"
server_key_iv: "158c38f657e01f347bd9c9e4d01c4b7f"
server_key_version: "encryptionkey@0"
I20260929 15:44:32.934454 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:32.950824 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:32.956018 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:32.956408 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:32.957579 2919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:32.959061 2919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:32.959136 2919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:32.959267 2919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:32.959332 2919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:32.997350 2919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33395
I20260929 15:44:32.997398 3046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33395 every 8 connection(s)
I20260929 15:44:32.998735 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:33.005543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 2919
I20260929 15:44:33.005765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:33.006619 388 external_mini_cluster.cc:1428] Setting key a1b52ac7bd954f916aab2ba7ea21acf4
Sep 29 15:44:33 dist-test-slave-7491 krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:33.023347 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:33.001433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33231 (local address 127.0.97.62:41969)
0929 15:44:33.001992 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:33.001998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:33.002774 (+ 776us) server_negotiation.cc:407] Connection header received
0929 15:44:33.004859 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:33.004864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:33.004950 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:33.005117 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:33.007425 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.008474 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.010215 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.010816 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.010991 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:33.015024 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:33.015032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:33.015039 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:33.015089 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:33.019167 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:33.020241 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:33.020250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:33.020256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:33.020421 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:33.020949 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:33.020957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:33.020963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:33.021365 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:33.021561 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:33.022554 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 15:44:33.022844 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":352,"threads_started":1}
I20260929 15:44:33.024065 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:33.001806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41969 (local address 127.0.97.1:33231)
0929 15:44:33.002602 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:33.002645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:33.004481 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:33.005455 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:33.005467 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:33.006295 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:33.007188 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:33.007217 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.008697 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.008702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:33.010032 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:33.010046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.011127 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.011135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:33.011461 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:33.012274 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:33.012299 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:33.014733 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:33.019395 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:33.019404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:33.019421 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:33.020060 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:33.020596 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:33.020600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:33.020603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:33.020815 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:33.021650 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:33.021666 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:33.022108 (+ 442us) client_negotiation.cc:769] Sending connection context
0929 15:44:33.022410 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 15:44:33.022854 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":413,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:44:33.026424 3047 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41969
I20260929 15:44:33.026772 3047 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:33.027724 3047 heartbeater.cc:511] Master 127.0.97.62:41969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:44:33.030843 2858 ts_manager.cc:194] Registered new tserver with Master: c26d05b6cad246b5bc01eff5838ccd51 (127.0.97.1:33395)
I20260929 15:44:33.033473 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:33.056704 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41969
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:33.422415 3053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:33.422757 3053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:33.422861 3053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:33.430677 3053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:33.430830 3053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:33.430950 3053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:33.441756 3053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:33.443178 3053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:33.444645 3053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:33.461123 3058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:33.461175 3059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:33.461145 3061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:33.462684 3053 server_base.cc:1034] running on GCE node
I20260929 15:44:33.463765 3053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:33.465507 3053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:33.466812 3053 hybrid_clock.cc:648] HybridClock initialized: now 1790696673466752 us; error 61 us; skew 500 ppm
I20260929 15:44:33.467365 3053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:33 dist-test-slave-7491 krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:33.471498 3053 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:33.474015 3053 webserver.cc:460] Webserver started at http://127.0.97.2:36333/ using document root <none> and password file <none>
I20260929 15:44:33.475039 3053 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:33.475183 3053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:33.475657 3053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:33.479758 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd81280b68a6465a88c3bbffd768969b"
format_stamp: "Formatted at 2026-09-29 15:44:33 on dist-test-slave-7491"
server_key: "07f6ed96e3ba5b02f12f0d682ee52b58"
server_key_iv: "530a193d2f2de2c2572eb4daecc6607b"
server_key_version: "encryptionkey@0"
I20260929 15:44:33.481325 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd81280b68a6465a88c3bbffd768969b"
format_stamp: "Formatted at 2026-09-29 15:44:33 on dist-test-slave-7491"
server_key: "07f6ed96e3ba5b02f12f0d682ee52b58"
server_key_iv: "530a193d2f2de2c2572eb4daecc6607b"
server_key_version: "encryptionkey@0"
I20260929 15:44:33.488325 3053 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:44:33.492656 3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:33.494416 3053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:44:33.494630 3053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "bd81280b68a6465a88c3bbffd768969b"
format_stamp: "Formatted at 2026-09-29 15:44:33 on dist-test-slave-7491"
server_key: "07f6ed96e3ba5b02f12f0d682ee52b58"
server_key_iv: "530a193d2f2de2c2572eb4daecc6607b"
server_key_version: "encryptionkey@0"
I20260929 15:44:33.494885 3053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:33.520674 3053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:33.525486 3053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:33.525803 3053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:33.526834 3053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:33.528281 3053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:33.528363 3053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:33.528488 3053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:33.528540 3053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:33.565480 3053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43197
I20260929 15:44:33.565590 3180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43197 every 8 connection(s)
I20260929 15:44:33.566874 3053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:33.571398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3053
I20260929 15:44:33.571589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:33.572137 388 external_mini_cluster.cc:1428] Setting key 2ddcc7bcc9907128db05274204cf0172
Sep 29 15:44:33 dist-test-slave-7491 krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:33.591610 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:33.569609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51121 (local address 127.0.97.62:41969)
0929 15:44:33.570186 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:33.570192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:33.571307 (+ 1115us) server_negotiation.cc:407] Connection header received
0929 15:44:33.573595 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:33.573600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:33.573684 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:33.573830 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:33.575714 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.576670 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.578372 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.578829 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.578969 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:33.583709 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:33.583728 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:33.583737 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:33.583792 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:33.587663 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:33.588637 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:33.588646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:33.588651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:33.588814 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:33.589369 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:33.589377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:33.589383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:33.589800 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:33.589965 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:33.590882 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 15:44:33.591171 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":346,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:44:33.592617 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:33.570317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41969 (local address 127.0.97.2:51121)
0929 15:44:33.571114 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:33.571159 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:33.573289 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:33.574094 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:33.574103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:33.574654 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:33.575433 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:33.575473 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.576856 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.576861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:33.578184 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:33.578198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:33.579093 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:33.579100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:33.579347 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:33.580354 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:33.580390 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:33.583288 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:33.587862 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:33.587871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:33.587886 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:33.588487 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:33.588983 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:33.588987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:33.588999 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:33.589222 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:33.589985 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:33.589995 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:33.590362 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 15:44:33.590607 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 15:44:33.591145 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":351,"threads_started":1}
I20260929 15:44:33.594327 3181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41969
I20260929 15:44:33.594676 3181 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:44:33.595521 3181 heartbeater.cc:511] Master 127.0.97.62:41969 requested a full tablet report, sending...
I20260929 15:44:33.597529 2858 ts_manager.cc:194] Registered new tserver with Master: bd81280b68a6465a88c3bbffd768969b (127.0.97.2:43197)
I20260929 15:44:33.598505 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:33.624713 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41969
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:44:34.038771 3047 heartbeater.cc:503] Master 127.0.97.62:41969 was elected leader, sending a full tablet report...
W20260929 15:44:34.050170 3187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:34.050649 3187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:34.050853 3187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:34.061537 3187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:34.061717 3187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:34.061959 3187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:34.075134 3187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:34.077250 3187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:34.078974 3187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:34.096033 3192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:34.096148 3193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:34.096195 3195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:34.097801 3187 server_base.cc:1034] running on GCE node
I20260929 15:44:34.098644 3187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:34.100504 3187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:34.101856 3187 hybrid_clock.cc:648] HybridClock initialized: now 1790696674101772 us; error 74 us; skew 500 ppm
I20260929 15:44:34.102303 3187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:34 dist-test-slave-7491 krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:34.106339 3187 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:34.108245 3187 webserver.cc:460] Webserver started at http://127.0.97.3:44953/ using document root <none> and password file <none>
I20260929 15:44:34.109051 3187 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:34.109216 3187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:34.109551 3187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:34.113160 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "f71a0ea333e041de9d4c358f25a54add"
format_stamp: "Formatted at 2026-09-29 15:44:34 on dist-test-slave-7491"
server_key: "fd1009c7349b54665a51edd0a5633b6d"
server_key_iv: "a7533b1dadfb8cc740a8fb21e8dbdcc6"
server_key_version: "encryptionkey@0"
I20260929 15:44:34.114274 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f71a0ea333e041de9d4c358f25a54add"
format_stamp: "Formatted at 2026-09-29 15:44:34 on dist-test-slave-7491"
server_key: "fd1009c7349b54665a51edd0a5633b6d"
server_key_iv: "a7533b1dadfb8cc740a8fb21e8dbdcc6"
server_key_version: "encryptionkey@0"
I20260929 15:44:34.121989 3187 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 15:44:34.126684 3202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:34.128616 3187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:44:34.128801 3187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "f71a0ea333e041de9d4c358f25a54add"
format_stamp: "Formatted at 2026-09-29 15:44:34 on dist-test-slave-7491"
server_key: "fd1009c7349b54665a51edd0a5633b6d"
server_key_iv: "a7533b1dadfb8cc740a8fb21e8dbdcc6"
server_key_version: "encryptionkey@0"
I20260929 15:44:34.129011 3187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:34.168680 3187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:34.174669 3187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:34.175020 3187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:34.176263 3187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:34.178138 3187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:34.178232 3187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:34.178329 3187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:34.178376 3187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:34.219313 3187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45919
I20260929 15:44:34.219466 3314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45919 every 8 connection(s)
I20260929 15:44:34.220911 3187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:34.231040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3187
I20260929 15:44:34.231298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:34.232012 388 external_mini_cluster.cc:1428] Setting key d73a23ed1eb17e4c707bc7fa8f491147
Sep 29 15:44:34 dist-test-slave-7491 krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:34.247066 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:34.224610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52627 (local address 127.0.97.62:41969)
0929 15:44:34.225113 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:34.225117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:34.226383 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 15:44:34.228483 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:34.228488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:34.228573 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:34.228718 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:34.230594 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:34.232159 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:34.235144 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:34.235677 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:34.235797 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:34.240445 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:34.240451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:34.240455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:34.240504 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:34.243338 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:34.244318 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:34.244323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:34.244326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:34.244468 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:34.245247 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:34.245251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:34.245253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:34.245554 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:34.245677 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:34.246573 (+ 896us) server_negotiation.cc:299] Negotiation successful
0929 15:44:34.246781 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260929 15:44:34.248253 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:34.225371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41969 (local address 127.0.97.3:52627)
0929 15:44:34.226184 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:34.226230 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:34.228000 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:34.228964 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:34.228974 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:34.229559 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:34.230240 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:34.230265 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:34.232569 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:34.232575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:34.234744 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:34.234761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:34.235977 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:34.235985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:34.236389 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:34.237181 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:34.237207 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:34.240065 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:34.243564 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:34.243572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:34.243591 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:34.244123 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:34.244672 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:34.244678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:34.244681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:34.245082 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:34.245708 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:34.245714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:34.246129 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 15:44:34.246390 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 15:44:34.246830 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":337,"threads_started":1}
I20260929 15:44:34.249805 3315 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41969
I20260929 15:44:34.250195 3315 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:34.251122 3315 heartbeater.cc:511] Master 127.0.97.62:41969 requested a full tablet report, sending...
I20260929 15:44:34.253273 2858 ts_manager.cc:194] Registered new tserver with Master: f71a0ea333e041de9d4c358f25a54add (127.0.97.3:45919)
I20260929 15:44:34.254207 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52627
I20260929 15:44:34.263411 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:44:34.301441 3317 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:44:34.298613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.0.97.62:41969)
0929 15:44:34.299016 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:34.299024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:34.299233 (+ 209us) server_negotiation.cc:407] Connection header received
0929 15:44:34.299481 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:34.299485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:34.299568 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:34.299688 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:34.301178 (+ 1490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41416: BlockingRecv error: recv got EOF from 127.0.0.1:41416 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
I20260929 15:44:34.304859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2919
W20260929 15:44:34.308291 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2919: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33395: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:44:34.308526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 2919
I20260929 15:44:34.324535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3053
W20260929 15:44:34.327713 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3053: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43197: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:44:34.328008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3053
I20260929 15:44:34.346590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3187
W20260929 15:44:34.350404 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3187: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45919: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:44:34.350613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3187
W20260929 15:44:34.350651 3329 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:44:34.347795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.0.97.3:45919)
0929 15:44:34.348704 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:34.348712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:34.348745 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:44:34.348822 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:34.348833 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:34.349072 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:34.349267 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:34.350221 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50128: BlockingRecv error: recv got EOF from 127.0.0.1:50128 (error 108)
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":572,"threads_started":1}
I20260929 15:44:34.367347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 2826
I20260929 15:44:34.368785 2888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:34.480365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 2826
2026-09-29T15:44:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4101 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:34 dist-test-slave-7491 krb5kdc[3333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:34 dist-test-slave-7491 krb5kdc[3333](info): set up 2 sockets
Sep 29 15:44:34 dist-test-slave-7491 krb5kdc[3333](info): commencing operation
krb5kdc: starting...
W20260929 15:44:35.633554 388 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:35 dist-test-slave-7491 krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:35.881740 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45157
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:36.275885 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:36.276386 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:36.276556 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:36.285092 3346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:36.285238 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:36.285322 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:36.285413 3346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:36.285487 3346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:36.296590 3346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:36.298105 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:36.299567 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:36.313373 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:36.313483 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:36.313728 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:36.314718 3346 server_base.cc:1034] running on GCE node
I20260929 15:44:36.316053 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:36.318341 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:36.319756 3346 hybrid_clock.cc:648] HybridClock initialized: now 1790696676319699 us; error 79 us; skew 500 ppm
I20260929 15:44:36.320472 3346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:36 dist-test-slave-7491 krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:36.324486 3346 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:36.326485 3346 webserver.cc:460] Webserver started at http://127.0.97.62:42807/ using document root <none> and password file <none>
I20260929 15:44:36.327298 3346 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:36.327491 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:36.327862 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:36.332537 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "c10298a1da584f1ea472122f4f07e77e"
format_stamp: "Formatted at 2026-09-29 15:44:36 on dist-test-slave-7491"
server_key: "357f9d4a6d10390253f913665df447ac"
server_key_iv: "da4a935c03a790a300366534ce6eb168"
server_key_version: "encryptionkey@0"
I20260929 15:44:36.334084 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "c10298a1da584f1ea472122f4f07e77e"
format_stamp: "Formatted at 2026-09-29 15:44:36 on dist-test-slave-7491"
server_key: "357f9d4a6d10390253f913665df447ac"
server_key_iv: "da4a935c03a790a300366534ce6eb168"
server_key_version: "encryptionkey@0"
I20260929 15:44:36.342235 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260929 15:44:36.346824 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:36.348851 3346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:36.349099 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "c10298a1da584f1ea472122f4f07e77e"
format_stamp: "Formatted at 2026-09-29 15:44:36 on dist-test-slave-7491"
server_key: "357f9d4a6d10390253f913665df447ac"
server_key_iv: "da4a935c03a790a300366534ce6eb168"
server_key_version: "encryptionkey@0"
I20260929 15:44:36.349349 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:36.375469 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:36.381319 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:36.381672 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:36.404460 3346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45157
I20260929 15:44:36.404486 3412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45157 every 8 connection(s)
I20260929 15:44:36.405962 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:36.409032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3346
I20260929 15:44:36.409337 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:36.410341 388 external_mini_cluster.cc:1428] Setting key 1f55b760473a132879d3394c77de6d86
I20260929 15:44:36.412448 3413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:36.427379 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e: Bootstrap starting.
Sep 29 15:44:36 dist-test-slave-7491 krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:36.432117 3413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:36.433640 3413 log.cc:824] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:36.438822 3413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e: No bootstrap required, opened a new log
I20260929 15:44:36.440584 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:36.413206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.0.97.62:45157)
0929 15:44:36.414235 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:36.414248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:36.414296 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:44:36.416311 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:36.416352 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:36.417190 (+ 838us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:36.417780 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:36.420054 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:36.421620 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:36.423232 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:36.424077 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:36.424380 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:36.429416 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:36.429426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:36.429447 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:36.429503 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:36.433584 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:36.434697 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:36.434712 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:36.434726 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:36.435091 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:36.435760 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:36.435766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:36.435769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:36.436834 (+ 1065us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:36.437544 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:36.437903 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 15:44:36.438775 (+ 872us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":528,"threads_started":1}
I20260929 15:44:36.445623 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:44:36.448791 3413 raft_consensus.cc:348] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10298a1da584f1ea472122f4f07e77e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45157 } }
I20260929 15:44:36.449219 3413 raft_consensus.cc:374] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:36.449353 3413 raft_consensus.cc:729] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10298a1da584f1ea472122f4f07e77e, State: Initialized, Role: FOLLOWER
I20260929 15:44:36.450071 3413 consensus_queue.cc:260] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10298a1da584f1ea472122f4f07e77e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45157 } }
I20260929 15:44:36.450332 3413 raft_consensus.cc:388] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:36.450467 3413 raft_consensus.cc:482] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:36.450665 3413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:36.454799 3413 raft_consensus.cc:504] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10298a1da584f1ea472122f4f07e77e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45157 } }
I20260929 15:44:36.456022 3413 leader_election.cc:302] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c10298a1da584f1ea472122f4f07e77e; no voters:
I20260929 15:44:36.457041 3413 leader_election.cc:288] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:36.457401 3419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:36.459223 3419 raft_consensus.cc:686] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [term 1 LEADER]: Becoming Leader. State: Replica: c10298a1da584f1ea472122f4f07e77e, State: Running, Role: LEADER
I20260929 15:44:36.460018 3419 consensus_queue.cc:237] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10298a1da584f1ea472122f4f07e77e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45157 } }
I20260929 15:44:36.460399 3413 sys_catalog.cc:564] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:36.463816 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c10298a1da584f1ea472122f4f07e77e. Latest consensus state: current_term: 1 leader_uuid: "c10298a1da584f1ea472122f4f07e77e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10298a1da584f1ea472122f4f07e77e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45157 } } }
I20260929 15:44:36.464303 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:36.465114 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c10298a1da584f1ea472122f4f07e77e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10298a1da584f1ea472122f4f07e77e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45157 } } }
I20260929 15:44:36.465473 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:36.476780 3425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:36.484741 3425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:36.502786 3425 catalog_manager.cc:1347] Generated new cluster ID: eb876af014d84749829f3006e49fe07c
I20260929 15:44:36.503046 3425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:36.526155 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:36.514901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.0.97.62:45157)
0929 15:44:36.515164 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:36.515173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:36.516062 (+ 889us) server_negotiation.cc:407] Connection header received
0929 15:44:36.516383 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:36.516390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:36.516502 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:36.516652 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:36.518082 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:36.519336 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:36.521755 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:36.522486 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:36.522652 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:36.524124 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:36.524132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:36.524138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:36.525169 (+ 1031us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:36.525246 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:36.525286 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:36.525431 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:36.525596 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 15:44:36.525720 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:44:36.536226 3425 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:36.538769 3425 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:36.553206 3425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c10298a1da584f1ea472122f4f07e77e: Generated new TSK 0
I20260929 15:44:36.554638 3425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:44:36.566476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3346
I20260929 15:44:36.568310 3408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:36.673684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3346
2026-09-29T15:44:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2186 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:36 dist-test-slave-7491 krb5kdc[3447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:36 dist-test-slave-7491 krb5kdc[3447](info): set up 2 sockets
Sep 29 15:44:36 dist-test-slave-7491 krb5kdc[3447](info): commencing operation
krb5kdc: starting...
W20260929 15:44:37.841356 388 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:37 dist-test-slave-7491 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:38.081849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37555
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:38.480291 3460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:38.480682 3460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:38.480787 3460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:38.489369 3460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:38.489527 3460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:38.489579 3460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:38.489634 3460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:38.489701 3460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:38.501072 3460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:38.503382 3460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:38.505038 3460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:38.519932 3468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:38.520071 3466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:38.520646 3465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:38.521054 3460 server_base.cc:1034] running on GCE node
I20260929 15:44:38.522004 3460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:38.523793 3460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:38.525130 3460 hybrid_clock.cc:648] HybridClock initialized: now 1790696678525087 us; error 82 us; skew 500 ppm
I20260929 15:44:38.525703 3460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:38 dist-test-slave-7491 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:38.535086 3460 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:38.537261 3460 webserver.cc:460] Webserver started at http://127.0.97.62:34203/ using document root <none> and password file <none>
I20260929 15:44:38.538546 3460 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:38.538679 3460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:38.539076 3460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:38.542527 3460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "cccd2c8c6eda4cfba02a0bd11beafc38"
format_stamp: "Formatted at 2026-09-29 15:44:38 on dist-test-slave-7491"
server_key: "02c407ff33c4afd5e23265b620eb79fd"
server_key_iv: "e45d4b25ab66a2e4dcccfc2ee942a55f"
server_key_version: "encryptionkey@0"
I20260929 15:44:38.543439 3460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "cccd2c8c6eda4cfba02a0bd11beafc38"
format_stamp: "Formatted at 2026-09-29 15:44:38 on dist-test-slave-7491"
server_key: "02c407ff33c4afd5e23265b620eb79fd"
server_key_iv: "e45d4b25ab66a2e4dcccfc2ee942a55f"
server_key_version: "encryptionkey@0"
I20260929 15:44:38.549381 3460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 15:44:38.553364 3475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:38.554951 3460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:44:38.555258 3460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "cccd2c8c6eda4cfba02a0bd11beafc38"
format_stamp: "Formatted at 2026-09-29 15:44:38 on dist-test-slave-7491"
server_key: "02c407ff33c4afd5e23265b620eb79fd"
server_key_iv: "e45d4b25ab66a2e4dcccfc2ee942a55f"
server_key_version: "encryptionkey@0"
I20260929 15:44:38.555470 3460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:38.570755 3460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:38.576740 3460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:38.577153 3460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:38.602632 3460 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37555
I20260929 15:44:38.602663 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37555 every 8 connection(s)
I20260929 15:44:38.604233 3460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:38.607450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3460
I20260929 15:44:38.607722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:38.608525 388 external_mini_cluster.cc:1428] Setting key 28ee2dd519ee85ffc8184f9c0ac153d7
I20260929 15:44:38.609195 3527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:44:38 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:38.627429 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38: Bootstrap starting.
I20260929 15:44:38.631299 3527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:38.632879 3527 log.cc:824] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:38.636940 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38: No bootstrap required, opened a new log
I20260929 15:44:38.638026 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:38.611127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.0.97.62:37555)
0929 15:44:38.612028 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:38.612042 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:38.612093 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:44:38.614276 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:38.614312 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:38.615112 (+ 800us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:38.615650 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:38.617698 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:38.619263 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:38.621204 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:38.621977 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:38.622142 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:38.627818 (+ 5676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:38.627831 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:38.627850 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:38.627909 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:38.631649 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:38.632853 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:38.632872 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:38.632883 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:38.633406 (+ 523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:38.633977 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:38.633994 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:38.633999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:38.634802 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:38.635072 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:38.635871 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 15:44:38.636613 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":432,"threads_started":1}
I20260929 15:44:38.645517 3527 raft_consensus.cc:348] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37555 } }
I20260929 15:44:38.645948 3527 raft_consensus.cc:374] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:38.646082 3527 raft_consensus.cc:729] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccd2c8c6eda4cfba02a0bd11beafc38, State: Initialized, Role: FOLLOWER
I20260929 15:44:38.646778 3527 consensus_queue.cc:260] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37555 } }
I20260929 15:44:38.647037 3527 raft_consensus.cc:388] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:38.647205 3527 raft_consensus.cc:482] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:38.647383 3527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:38.649909 3527 raft_consensus.cc:504] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37555 } }
I20260929 15:44:38.650552 3527 leader_election.cc:302] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cccd2c8c6eda4cfba02a0bd11beafc38; no voters:
I20260929 15:44:38.651252 3527 leader_election.cc:288] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:38.651434 3532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:38.652298 3532 raft_consensus.cc:686] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [term 1 LEADER]: Becoming Leader. State: Replica: cccd2c8c6eda4cfba02a0bd11beafc38, State: Running, Role: LEADER
I20260929 15:44:38.653021 3532 consensus_queue.cc:237] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37555 } }
I20260929 15:44:38.654167 3527 sys_catalog.cc:564] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:38.656339 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37555 } } }
I20260929 15:44:38.657160 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:38.656333 3534 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cccd2c8c6eda4cfba02a0bd11beafc38. Latest consensus state: current_term: 1 leader_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccd2c8c6eda4cfba02a0bd11beafc38" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37555 } } }
I20260929 15:44:38.660092 3534 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:38.665464 3538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:38.670711 3538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:38.682752 3538 catalog_manager.cc:1347] Generated new cluster ID: 1c4f2a9e8dad433e8d5baf279da96cbe
I20260929 15:44:38.682919 3538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:38.707459 3538 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:38.709592 3538 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:38.727149 3538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cccd2c8c6eda4cfba02a0bd11beafc38: Generated new TSK 0
I20260929 15:44:38.728405 3538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:38.789168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37555
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:39.202950 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:39.203378 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:39.203600 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:39.212576 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:39.212736 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:39.212936 3553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:39.224252 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:39.225864 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:39.227622 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:39.243860 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:39.243824 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:39.243906 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:39.245067 3553 server_base.cc:1034] running on GCE node
I20260929 15:44:39.245846 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:39.246990 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:39.250499 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790696679250353 us; error 127 us; skew 500 ppm
I20260929 15:44:39.251016 3553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:39 dist-test-slave-7491 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:39.255308 3553 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:39.257231 3553 webserver.cc:460] Webserver started at http://127.0.97.1:36997/ using document root <none> and password file <none>
I20260929 15:44:39.258060 3553 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:39.258227 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:39.258562 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:39.263504 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "bf41b97804d64579921e722c535c8c08"
format_stamp: "Formatted at 2026-09-29 15:44:39 on dist-test-slave-7491"
server_key: "688dd64a882bb28ba99471d17a9688a2"
server_key_iv: "2ba331d8bf58a5828a2168c0f4c5485d"
server_key_version: "encryptionkey@0"
I20260929 15:44:39.264696 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bf41b97804d64579921e722c535c8c08"
format_stamp: "Formatted at 2026-09-29 15:44:39 on dist-test-slave-7491"
server_key: "688dd64a882bb28ba99471d17a9688a2"
server_key_iv: "2ba331d8bf58a5828a2168c0f4c5485d"
server_key_version: "encryptionkey@0"
I20260929 15:44:39.272395 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 15:44:39.277310 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:39.279241 3553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:44:39.279592 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "bf41b97804d64579921e722c535c8c08"
format_stamp: "Formatted at 2026-09-29 15:44:39 on dist-test-slave-7491"
server_key: "688dd64a882bb28ba99471d17a9688a2"
server_key_iv: "2ba331d8bf58a5828a2168c0f4c5485d"
server_key_version: "encryptionkey@0"
I20260929 15:44:39.279870 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:39.302129 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:39.308547 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:39.309002 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:39.310515 3553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:39.312642 3553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:39.312784 3553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:39.312970 3553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:39.313061 3553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:39.355934 3553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45907
I20260929 15:44:39.356027 3680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45907 every 8 connection(s)
I20260929 15:44:39.357875 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:39.365849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3553
I20260929 15:44:39.366391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:39.367383 388 external_mini_cluster.cc:1428] Setting key 42a7fc60a20198a183be5bfb50bca288
Sep 29 15:44:39 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:39.386826 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:39.361841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53015 (local address 127.0.97.62:37555)
0929 15:44:39.362424 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:39.362428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:39.363934 (+ 1506us) server_negotiation.cc:407] Connection header received
0929 15:44:39.366827 (+ 2893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:39.366833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:39.366946 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:39.367119 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:39.369172 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.370307 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.372989 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.373470 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.373653 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:39.379176 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:39.379182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:39.379186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:39.379229 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:39.382166 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:39.383188 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:39.383249 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:39.383289 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:39.383550 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:39.384374 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:39.384383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:39.384387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:39.384758 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:39.385008 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:39.386046 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 15:44:39.386361 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:44:39.388398 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:39.362631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37555 (local address 127.0.97.1:53015)
0929 15:44:39.363701 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:39.363763 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:39.366369 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:39.367486 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:39.367504 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:39.368266 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:39.368964 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:39.368985 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.370659 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.370665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:39.372708 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:39.372731 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.373968 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.374005 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:39.374383 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:39.375320 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:39.375353 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:39.378801 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:39.382397 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:39.382410 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:39.382440 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:39.382979 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:39.383809 (+ 830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:39.383816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:39.383820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:39.384147 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:39.384939 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:39.384950 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:39.385565 (+ 615us) client_negotiation.cc:769] Sending connection context
0929 15:44:39.385918 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 15:44:39.386699 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":400,"threads_started":1}
I20260929 15:44:39.391203 3681 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:44:39.391803 3681 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:39.393167 3681 heartbeater.cc:511] Master 127.0.97.62:37555 requested a full tablet report, sending...
I20260929 15:44:39.396376 3492 ts_manager.cc:194] Registered new tserver with Master: bf41b97804d64579921e722c535c8c08 (127.0.97.1:45907)
I20260929 15:44:39.398319 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:39.421051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37555
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:39.807780 3687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:39.808104 3687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:39.808311 3687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:39.818394 3687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:39.818562 3687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:39.818902 3687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:39.830684 3687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:39.832238 3687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:39.833835 3687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:39.848913 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:39.848937 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:39.848871 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:39.850124 3687 server_base.cc:1034] running on GCE node
I20260929 15:44:39.851189 3687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:39.852712 3687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:39.854094 3687 hybrid_clock.cc:648] HybridClock initialized: now 1790696679854000 us; error 64 us; skew 500 ppm
I20260929 15:44:39.854863 3687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:39 dist-test-slave-7491 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:39.859413 3687 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:39.861644 3687 webserver.cc:460] Webserver started at http://127.0.97.2:36973/ using document root <none> and password file <none>
I20260929 15:44:39.862560 3687 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:39.862685 3687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:39.863010 3687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:39.867162 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "a62eea3198e34d5592653929977e3879"
format_stamp: "Formatted at 2026-09-29 15:44:39 on dist-test-slave-7491"
server_key: "07f40cc72700ae15e876b3ef51c7942b"
server_key_iv: "d9de306d09cc8c744a3ccb2209e7d1d2"
server_key_version: "encryptionkey@0"
I20260929 15:44:39.868111 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a62eea3198e34d5592653929977e3879"
format_stamp: "Formatted at 2026-09-29 15:44:39 on dist-test-slave-7491"
server_key: "07f40cc72700ae15e876b3ef51c7942b"
server_key_iv: "d9de306d09cc8c744a3ccb2209e7d1d2"
server_key_version: "encryptionkey@0"
I20260929 15:44:39.875520 3687 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 15:44:39.880121 3702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:39.882054 3687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 15:44:39.882296 3687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "a62eea3198e34d5592653929977e3879"
format_stamp: "Formatted at 2026-09-29 15:44:39 on dist-test-slave-7491"
server_key: "07f40cc72700ae15e876b3ef51c7942b"
server_key_iv: "d9de306d09cc8c744a3ccb2209e7d1d2"
server_key_version: "encryptionkey@0"
I20260929 15:44:39.882541 3687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:39.903483 3687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:39.908586 3687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:39.908955 3687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:39.910096 3687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:39.912783 3687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:39.912909 3687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:39.913084 3687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:39.913154 3687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:39.956745 3687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45667
I20260929 15:44:39.956779 3814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45667 every 8 connection(s)
I20260929 15:44:39.958644 3687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:39.965801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3687
I20260929 15:44:39.966029 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:39.966563 388 external_mini_cluster.cc:1428] Setting key 2dde26ed0d2a843fc25c99c57bedbe01
Sep 29 15:44:39 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:39.989764 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:39.962001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34793 (local address 127.0.97.62:37555)
0929 15:44:39.962491 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:39.962496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:39.964033 (+ 1537us) server_negotiation.cc:407] Connection header received
0929 15:44:39.967163 (+ 3130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:39.967169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:39.967275 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:39.967422 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:39.970026 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.970865 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.973737 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.974265 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.974419 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:39.980055 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:39.980067 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:39.980076 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:39.980141 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:39.984764 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:39.986358 (+ 1594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:39.986367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:39.986372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:39.986617 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:39.987343 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:39.987350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:39.987354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:39.987736 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:39.987933 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:39.989014 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0929 15:44:39.989337 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:44:39.991110 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:39.962730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37555 (local address 127.0.97.2:34793)
0929 15:44:39.963789 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:39.963842 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:39.966756 (+ 2914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:39.967798 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:39.967812 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:39.968770 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:39.969764 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:39.969801 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.971065 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.971075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:39.973462 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:39.973484 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:39.974670 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:39.974683 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:39.975153 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:39.976225 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:39.976272 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:39.979599 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:39.985199 (+ 5600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:39.985210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:39.985227 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:39.986125 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:39.986782 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:39.986786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:39.986789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:39.987110 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:39.987894 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:39.987903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:39.988551 (+ 648us) client_negotiation.cc:769] Sending connection context
0929 15:44:39.988885 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 15:44:39.989525 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":409,"threads_started":1}
I20260929 15:44:39.993084 3815 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:44:39.993490 3815 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:39.994406 3815 heartbeater.cc:511] Master 127.0.97.62:37555 requested a full tablet report, sending...
I20260929 15:44:39.996526 3492 ts_manager.cc:194] Registered new tserver with Master: a62eea3198e34d5592653929977e3879 (127.0.97.2:45667)
I20260929 15:44:39.998052 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:40.023872 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37555
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:40.389151 3821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:40.389552 3821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:40.389748 3821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:40.397701 3821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:40.397890 3821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:40.398080 3821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:40.402248 3681 heartbeater.cc:503] Master 127.0.97.62:37555 was elected leader, sending a full tablet report...
I20260929 15:44:40.409103 3821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:40.410635 3821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:40.412088 3821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:40.428020 3827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:40.428108 3829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:40.428058 3826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:40.429280 3821 server_base.cc:1034] running on GCE node
I20260929 15:44:40.430328 3821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:40.431993 3821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:40.433385 3821 hybrid_clock.cc:648] HybridClock initialized: now 1790696680433372 us; error 140 us; skew 500 ppm
I20260929 15:44:40.433965 3821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:40.438189 3821 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:40.440073 3821 webserver.cc:460] Webserver started at http://127.0.97.3:36945/ using document root <none> and password file <none>
I20260929 15:44:40.440938 3821 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:40.441115 3821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:40.441445 3821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:40.444877 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "7e4004ec3b7f4452999848066b6d7533"
format_stamp: "Formatted at 2026-09-29 15:44:40 on dist-test-slave-7491"
server_key: "d705faac52c836f08ba4ca614fa5c46e"
server_key_iv: "b10409b66c48b5b219aaa0d393076203"
server_key_version: "encryptionkey@0"
I20260929 15:44:40.445816 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7e4004ec3b7f4452999848066b6d7533"
format_stamp: "Formatted at 2026-09-29 15:44:40 on dist-test-slave-7491"
server_key: "d705faac52c836f08ba4ca614fa5c46e"
server_key_iv: "b10409b66c48b5b219aaa0d393076203"
server_key_version: "encryptionkey@0"
I20260929 15:44:40.452349 3821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 15:44:40.456619 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:40.458958 3821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:44:40.459172 3821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "7e4004ec3b7f4452999848066b6d7533"
format_stamp: "Formatted at 2026-09-29 15:44:40 on dist-test-slave-7491"
server_key: "d705faac52c836f08ba4ca614fa5c46e"
server_key_iv: "b10409b66c48b5b219aaa0d393076203"
server_key_version: "encryptionkey@0"
I20260929 15:44:40.459421 3821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:40.485507 3821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:40.491719 3821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:40.492161 3821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:40.493826 3821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:40.495584 3821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:40.495677 3821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:40.495780 3821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:40.495819 3821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:40.534487 3821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38493
I20260929 15:44:40.534793 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38493 every 8 connection(s)
I20260929 15:44:40.535954 3821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:40.540267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3821
I20260929 15:44:40.540482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:40.540948 388 external_mini_cluster.cc:1428] Setting key fd2fd08678e21cdaa18ee04b658fee44
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:40.561151 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:40.538803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40775 (local address 127.0.97.62:37555)
0929 15:44:40.539295 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:40.539300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:40.540491 (+ 1191us) server_negotiation.cc:407] Connection header received
0929 15:44:40.542700 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:40.542707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:40.542800 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:40.542931 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:40.544693 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.545506 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.547300 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.547752 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.547912 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:40.552981 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:40.552986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:40.552990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:40.553032 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:40.556755 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.557829 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.557836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.557840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.558033 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.558536 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.558542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.558545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.558838 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:40.559016 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:40.560437 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0929 15:44:40.560761 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":282,"threads_started":1}
I20260929 15:44:40.562224 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:40.539224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37555 (local address 127.0.97.3:40775)
0929 15:44:40.540297 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:40.540335 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:40.542386 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:40.543200 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:40.543209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:40.543778 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:40.544484 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:40.544507 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.545714 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.545720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:40.547102 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:40.547117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.548025 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.548033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:40.548387 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:40.549333 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:40.549373 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:40.552655 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:40.556949 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:40.556970 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:40.556988 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:40.557680 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:40.558168 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:40.558173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:40.558184 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:40.558396 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:40.558984 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:40.558995 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:40.559431 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 15:44:40.560254 (+ 823us) client_negotiation.cc:240] Negotiation successful
0929 15:44:40.560924 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":376,"threads_started":1}
I20260929 15:44:40.563947 3949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37555
I20260929 15:44:40.564400 3949 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:40.565274 3949 heartbeater.cc:511] Master 127.0.97.62:37555 requested a full tablet report, sending...
I20260929 15:44:40.567055 3492 ts_manager.cc:194] Registered new tserver with Master: 7e4004ec3b7f4452999848066b6d7533 (127.0.97.3:38493)
I20260929 15:44:40.567979 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40775
I20260929 15:44:40.571118 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:40.628176 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:40.609930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.0.97.62:37555)
0929 15:44:40.610203 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:40.610209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:40.610694 (+ 485us) server_negotiation.cc:407] Connection header received
0929 15:44:40.610977 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:40.610984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:40.611112 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:40.611357 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:40.613243 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.614071 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.615689 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.616275 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.616427 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:40.621679 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:40.621687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:40.621691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:40.621740 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:40.624792 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.625779 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.625785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.625788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.625952 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.626576 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.626580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.626582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.626982 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:40.627176 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:40.627674 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 15:44:40.627884 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260929 15:44:40.628804 3492 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46000
I20260929 15:44:40.630786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3553
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:40.651502 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:40.632766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.0.97.1:45907)
0929 15:44:40.633436 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:40.633444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:40.633472 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:44:40.633537 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:40.633544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:40.633779 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:40.633921 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:40.635784 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.637193 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.638789 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.639441 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.639638 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:40.644267 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:40.644288 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:40.644311 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:40.644354 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:40.647494 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.648412 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.648422 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.648424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.648559 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.649131 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.649139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.649148 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.649760 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:40.650157 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:40.650780 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 15:44:40.651105 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":354,"threads_started":1}
W20260929 15:44:40.653364 3676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55420
W20260929 15:44:40.655841 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:40.656105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3553
I20260929 15:44:40.673604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3687
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:40.693729 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:40.674615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.0.97.2:45667)
0929 15:44:40.675343 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:40.675350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:40.675379 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:44:40.675448 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:40.675455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:40.675670 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:40.675838 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:40.677434 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.678496 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.680365 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.681260 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.681407 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:40.686103 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:40.686128 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:40.686146 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:40.686191 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:40.689767 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.690822 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.690830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.690833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.691027 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.691702 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.691706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.691709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.692354 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:40.692653 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:40.693210 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 15:44:40.693434 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":458,"threads_started":1}
W20260929 15:44:40.695840 3810 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38002
W20260929 15:44:40.697685 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:40.698030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3687
I20260929 15:44:40.714645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3821
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696680, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:40.736472 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:40.716375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.0.97.3:38493)
0929 15:44:40.717427 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:40.717437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:40.717478 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:44:40.717550 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:40.717557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:40.717831 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:40.718086 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:40.720011 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.721336 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.722787 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:40.723400 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:40.723551 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:40.728245 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:40.728262 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:40.728288 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:40.728332 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:40.731985 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.733140 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.733149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.733152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.733363 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:40.734054 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:40.734060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:40.734063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:40.734698 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:40.735030 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:40.735557 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 15:44:40.736022 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":449,"threads_started":1}
W20260929 15:44:40.738348 3944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55554
W20260929 15:44:40.740010 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3821: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:44:40.740368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3821
I20260929 15:44:40.756328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3460
I20260929 15:44:40.758059 3522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:40.872423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3460
2026-09-29T15:44:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4204 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3967](info): set up 2 sockets
Sep 29 15:44:40 dist-test-slave-7491 krb5kdc[3967](info): commencing operation
krb5kdc: starting...
W20260929 15:44:42.037313 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:42 dist-test-slave-7491 krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:42.266028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45861
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:42.619555 3980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:42.619916 3980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:42.620103 3980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:42.627866 3980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:42.628003 3980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:42.628051 3980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:42.628109 3980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:42.628229 3980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:42.639195 3980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:42.640645 3980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:42.642027 3980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:42.655721 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:42.655973 3988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:42.655591 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:42.658201 3980 server_base.cc:1034] running on GCE node
I20260929 15:44:42.659518 3980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:42.661720 3980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:42.663103 3980 hybrid_clock.cc:648] HybridClock initialized: now 1790696682662995 us; error 87 us; skew 500 ppm
I20260929 15:44:42.663735 3980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:42 dist-test-slave-7491 krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:42.667894 3980 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:42.669893 3980 webserver.cc:460] Webserver started at http://127.0.97.62:37259/ using document root <none> and password file <none>
I20260929 15:44:42.670753 3980 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:42.670928 3980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:42.671277 3980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:42.674737 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "d497b41c9d3d43f48c0f63651bb2a750"
format_stamp: "Formatted at 2026-09-29 15:44:42 on dist-test-slave-7491"
server_key: "48c701fc453e194decfd5623dc677eb6"
server_key_iv: "4cfde50d35443f6c60033edf8450080e"
server_key_version: "encryptionkey@0"
I20260929 15:44:42.675686 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "d497b41c9d3d43f48c0f63651bb2a750"
format_stamp: "Formatted at 2026-09-29 15:44:42 on dist-test-slave-7491"
server_key: "48c701fc453e194decfd5623dc677eb6"
server_key_iv: "4cfde50d35443f6c60033edf8450080e"
server_key_version: "encryptionkey@0"
I20260929 15:44:42.683259 3980 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.006s
I20260929 15:44:42.688318 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:42.690462 3980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:42.690757 3980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "d497b41c9d3d43f48c0f63651bb2a750"
format_stamp: "Formatted at 2026-09-29 15:44:42 on dist-test-slave-7491"
server_key: "48c701fc453e194decfd5623dc677eb6"
server_key_iv: "4cfde50d35443f6c60033edf8450080e"
server_key_version: "encryptionkey@0"
I20260929 15:44:42.691012 3980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:42.709981 3980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:42.716028 3980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:42.716462 3980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:42.740520 3980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45861
I20260929 15:44:42.740561 4046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45861 every 8 connection(s)
I20260929 15:44:42.742372 3980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:42.748128 4047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:42.749987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 3980
I20260929 15:44:42.750252 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:42.750922 388 external_mini_cluster.cc:1428] Setting key 62ed2bd66f143367c6d77c09f64d549c
I20260929 15:44:42.764782 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750: Bootstrap starting.
I20260929 15:44:42.768421 4047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750: Neither blocks nor log segments found. Creating new log.
Sep 29 15:44:42 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:42.769593 4047 log.cc:824] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:42.773939 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750: No bootstrap required, opened a new log
I20260929 15:44:42.780802 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:42.753371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.0.97.62:45861)
0929 15:44:42.754377 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:42.754397 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:42.754458 (+ 61us) server_negotiation.cc:407] Connection header received
0929 15:44:42.756399 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:42.756446 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:42.757127 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:42.757670 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:42.759849 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:42.762061 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:42.764160 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:42.765358 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:42.765599 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:42.770173 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:42.770187 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:42.770213 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:42.770285 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:42.774598 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:42.775869 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:42.775885 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:42.775896 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:42.776199 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:42.776727 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:42.776736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:42.776742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:42.777570 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:42.777918 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:42.778636 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 15:44:42.779156 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":373,"threads_started":1}
I20260929 15:44:42.784462 4047 raft_consensus.cc:348] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d497b41c9d3d43f48c0f63651bb2a750" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260929 15:44:42.784865 4047 raft_consensus.cc:374] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:42.784938 4047 raft_consensus.cc:729] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d497b41c9d3d43f48c0f63651bb2a750, State: Initialized, Role: FOLLOWER
I20260929 15:44:42.785537 4047 consensus_queue.cc:260] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d497b41c9d3d43f48c0f63651bb2a750" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260929 15:44:42.785745 4047 raft_consensus.cc:388] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:42.785840 4047 raft_consensus.cc:482] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:42.785967 4047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:42.788349 4047 raft_consensus.cc:504] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d497b41c9d3d43f48c0f63651bb2a750" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260929 15:44:42.789007 4047 leader_election.cc:302] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d497b41c9d3d43f48c0f63651bb2a750; no voters:
I20260929 15:44:42.789783 4047 leader_election.cc:288] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:42.790032 4052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:42.790773 4052 raft_consensus.cc:686] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [term 1 LEADER]: Becoming Leader. State: Replica: d497b41c9d3d43f48c0f63651bb2a750, State: Running, Role: LEADER
I20260929 15:44:42.791460 4052 consensus_queue.cc:237] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d497b41c9d3d43f48c0f63651bb2a750" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } }
I20260929 15:44:42.792052 4047 sys_catalog.cc:564] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:42.794655 4053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d497b41c9d3d43f48c0f63651bb2a750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d497b41c9d3d43f48c0f63651bb2a750" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } } }
I20260929 15:44:42.794822 4054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d497b41c9d3d43f48c0f63651bb2a750. Latest consensus state: current_term: 1 leader_uuid: "d497b41c9d3d43f48c0f63651bb2a750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d497b41c9d3d43f48c0f63651bb2a750" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45861 } } }
I20260929 15:44:42.795115 4053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:42.795210 4054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:42.799474 4058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:42.804706 4058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:42.817034 4058 catalog_manager.cc:1347] Generated new cluster ID: 166dba1925e9493eb1b665ee896731b2
I20260929 15:44:42.817154 4058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:42.851105 4058 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:42.853147 4058 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:42.860066 4058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d497b41c9d3d43f48c0f63651bb2a750: Generated new TSK 0
I20260929 15:44:42.861112 4058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:42.931782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:43.362485 4073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:43.362798 4073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:43.362977 4073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:43.372645 4073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:43.372802 4073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:43.373031 4073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:43.384459 4073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:43.386050 4073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:43.387517 4073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:43.403702 4081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:43.403893 4078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:43.404036 4079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:43.405469 4073 server_base.cc:1034] running on GCE node
I20260929 15:44:43.406539 4073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:43.408301 4073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:43.409657 4073 hybrid_clock.cc:648] HybridClock initialized: now 1790696683409593 us; error 50 us; skew 500 ppm
I20260929 15:44:43.410243 4073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:43 dist-test-slave-7491 krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:43.414414 4073 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:43.416250 4073 webserver.cc:460] Webserver started at http://127.0.97.1:45115/ using document root <none> and password file <none>
I20260929 15:44:43.417006 4073 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:43.417157 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:43.417483 4073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:43.420670 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "775b74480d794d7fb56d4c8c1f99766b"
format_stamp: "Formatted at 2026-09-29 15:44:43 on dist-test-slave-7491"
server_key: "1482b1381f9aa32f4e4ab0dd834a31b9"
server_key_iv: "91fb5646e848335624ebb789e4ebf9dd"
server_key_version: "encryptionkey@0"
I20260929 15:44:43.421499 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "775b74480d794d7fb56d4c8c1f99766b"
format_stamp: "Formatted at 2026-09-29 15:44:43 on dist-test-slave-7491"
server_key: "1482b1381f9aa32f4e4ab0dd834a31b9"
server_key_iv: "91fb5646e848335624ebb789e4ebf9dd"
server_key_version: "encryptionkey@0"
I20260929 15:44:43.427497 4073 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:43.431592 4088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:43.433364 4073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 15:44:43.433589 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "775b74480d794d7fb56d4c8c1f99766b"
format_stamp: "Formatted at 2026-09-29 15:44:43 on dist-test-slave-7491"
server_key: "1482b1381f9aa32f4e4ab0dd834a31b9"
server_key_iv: "91fb5646e848335624ebb789e4ebf9dd"
server_key_version: "encryptionkey@0"
I20260929 15:44:43.433825 4073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:43.449963 4073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:43.455772 4073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:43.456462 4073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:43.457790 4073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:43.459400 4073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:43.459522 4073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:43.459690 4073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:43.459766 4073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:43.502697 4073 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34107
I20260929 15:44:43.502744 4200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34107 every 8 connection(s)
I20260929 15:44:43.504231 4073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:43.509811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4073
I20260929 15:44:43.510030 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:43.510850 388 external_mini_cluster.cc:1428] Setting key 3ea89b1235b0890564609af7a9601b93
Sep 29 15:44:43 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:43.532212 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:43.507963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46749 (local address 127.0.97.62:45861)
0929 15:44:43.508874 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:43.508883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:43.510272 (+ 1389us) server_negotiation.cc:407] Connection header received
0929 15:44:43.512860 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:43.512866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:43.512952 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:43.513094 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:43.515430 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:43.516289 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:43.519095 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:43.519534 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:43.519767 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:43.523909 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:43.523919 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:43.523927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:43.523977 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:43.527974 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:43.529214 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:43.529222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:43.529226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:43.529399 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:43.529951 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:43.529959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:43.529965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:43.530408 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:43.530598 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:43.531445 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 15:44:43.531725 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":433,"threads_started":1}
I20260929 15:44:43.533046 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:43.508997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45861 (local address 127.0.97.1:46749)
0929 15:44:43.510023 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:43.510114 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:43.512546 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:43.513390 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:43.513399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:43.514223 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:43.515180 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:43.515212 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:43.516464 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:43.516474 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:43.518821 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:43.518844 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:43.519894 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:43.519900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:43.520127 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:43.520979 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:43.521004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:43.523604 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:43.528313 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:43.528324 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:43.528342 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:43.529046 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:43.529581 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:43.529585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:43.529587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:43.529803 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:43.530570 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:43.530577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:43.531005 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 15:44:43.531270 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 15:44:43.531723 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":376,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:44:43.534958 4201 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45861
I20260929 15:44:43.535317 4201 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:44:43.536226 4201 heartbeater.cc:511] Master 127.0.97.62:45861 requested a full tablet report, sending...
I20260929 15:44:43.539175 4012 ts_manager.cc:194] Registered new tserver with Master: 775b74480d794d7fb56d4c8c1f99766b (127.0.97.1:34107)
I20260929 15:44:43.540966 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:43.565085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:43.960009 4207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:43.960443 4207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:43.960633 4207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:43.968346 4207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:43.968482 4207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:43.968665 4207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:43.978888 4207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:43.980358 4207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:43.981853 4207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:43.996932 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:43.996860 4212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:43.996904 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:43.998692 4207 server_base.cc:1034] running on GCE node
I20260929 15:44:43.999758 4207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:44.001340 4207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:44.002651 4207 hybrid_clock.cc:648] HybridClock initialized: now 1790696684002616 us; error 73 us; skew 500 ppm
I20260929 15:44:44.003077 4207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:44.011487 4207 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:44.013819 4207 webserver.cc:460] Webserver started at http://127.0.97.2:39383/ using document root <none> and password file <none>
I20260929 15:44:44.014814 4207 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:44.014998 4207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:44.015362 4207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:44.018630 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "ac0f8aebcc474ae6a0ba949a2c655fe9"
format_stamp: "Formatted at 2026-09-29 15:44:44 on dist-test-slave-7491"
server_key: "ed4b87418d4db7e8cfaeead3fb50d0a5"
server_key_iv: "66541c623c39870ad07f37524691203a"
server_key_version: "encryptionkey@0"
I20260929 15:44:44.019544 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ac0f8aebcc474ae6a0ba949a2c655fe9"
format_stamp: "Formatted at 2026-09-29 15:44:44 on dist-test-slave-7491"
server_key: "ed4b87418d4db7e8cfaeead3fb50d0a5"
server_key_iv: "66541c623c39870ad07f37524691203a"
server_key_version: "encryptionkey@0"
I20260929 15:44:44.025686 4207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260929 15:44:44.030993 4222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:44.033509 4207 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 15:44:44.033792 4207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "ac0f8aebcc474ae6a0ba949a2c655fe9"
format_stamp: "Formatted at 2026-09-29 15:44:44 on dist-test-slave-7491"
server_key: "ed4b87418d4db7e8cfaeead3fb50d0a5"
server_key_iv: "66541c623c39870ad07f37524691203a"
server_key_version: "encryptionkey@0"
I20260929 15:44:44.034070 4207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:44.073760 4207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:44.079764 4207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:44.080346 4207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:44.081440 4207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:44.083140 4207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:44.083266 4207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:44.083442 4207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:44.083518 4207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:44.128461 4207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43573
I20260929 15:44:44.128527 4334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43573 every 8 connection(s)
I20260929 15:44:44.129998 4207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:44.133884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4207
I20260929 15:44:44.134076 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:44.134608 388 external_mini_cluster.cc:1428] Setting key c761ad6ba7679dc2e584c0f9d17afa8f
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:44:44.158655 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.133733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36879 (local address 127.0.97.62:45861)
0929 15:44:44.134327 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:44.134335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:44.135983 (+ 1648us) server_negotiation.cc:407] Connection header received
0929 15:44:44.139119 (+ 3136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:44.139128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:44.139232 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:44.139373 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:44.141750 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.142597 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.144465 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.144969 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.145119 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.150526 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:44.150534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:44.150538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:44.150588 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:44.153976 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.155451 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.155456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.155459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.155611 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.156415 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.156428 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.156430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.156895 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:44.157079 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:44.158047 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 15:44:44.158290 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":324,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:44:44.159845 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.134532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45861 (local address 127.0.97.2:36879)
0929 15:44:44.135793 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:44.135848 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:44.138644 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:44.139906 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:44.139919 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:44.140731 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:44.141514 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:44.141537 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.142794 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.142799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:44.144309 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:44.144323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.145238 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.145246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:44.145570 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.146794 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:44.146833 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:44.150029 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:44.154281 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:44.154299 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:44.154329 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:44.155214 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:44.155813 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:44.155821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:44.155827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:44.156158 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:44.157081 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:44.157089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:44.157486 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 15:44:44.157834 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 15:44:44.158303 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":471,"threads_started":1}
I20260929 15:44:44.161852 4335 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45861
I20260929 15:44:44.162258 4335 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:44.163269 4335 heartbeater.cc:511] Master 127.0.97.62:45861 requested a full tablet report, sending...
I20260929 15:44:44.166173 4012 ts_manager.cc:194] Registered new tserver with Master: ac0f8aebcc474ae6a0ba949a2c655fe9 (127.0.97.2:43573)
I20260929 15:44:44.167089 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:44.189554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:44:44.545315 4201 heartbeater.cc:503] Master 127.0.97.62:45861 was elected leader, sending a full tablet report...
W20260929 15:44:44.586720 4341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:44.587042 4341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:44.587137 4341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:44.595724 4341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:44.595851 4341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:44.595969 4341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:44.606734 4341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:44.608114 4341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:44.609483 4341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:44.626425 4349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:44.626374 4346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:44.626466 4347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:44.630466 4341 server_base.cc:1034] running on GCE node
I20260929 15:44:44.631279 4341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:44.632534 4341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:44.633816 4341 hybrid_clock.cc:648] HybridClock initialized: now 1790696684633751 us; error 61 us; skew 500 ppm
I20260929 15:44:44.634460 4341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:44.639012 4341 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:44.641394 4341 webserver.cc:460] Webserver started at http://127.0.97.3:36505/ using document root <none> and password file <none>
I20260929 15:44:44.642375 4341 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:44.642524 4341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:44.642916 4341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:44.647316 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "a94d9ff8664646068282972b2a60e89e"
format_stamp: "Formatted at 2026-09-29 15:44:44 on dist-test-slave-7491"
server_key: "1b002204dcb59e06a195ab5c413ac85e"
server_key_iv: "ec44c9357b3a31fa4470808389db27a6"
server_key_version: "encryptionkey@0"
I20260929 15:44:44.648397 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a94d9ff8664646068282972b2a60e89e"
format_stamp: "Formatted at 2026-09-29 15:44:44 on dist-test-slave-7491"
server_key: "1b002204dcb59e06a195ab5c413ac85e"
server_key_iv: "ec44c9357b3a31fa4470808389db27a6"
server_key_version: "encryptionkey@0"
I20260929 15:44:44.655015 4341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:44.660560 4356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:44.662772 4341 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:44:44.663218 4341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "a94d9ff8664646068282972b2a60e89e"
format_stamp: "Formatted at 2026-09-29 15:44:44 on dist-test-slave-7491"
server_key: "1b002204dcb59e06a195ab5c413ac85e"
server_key_iv: "ec44c9357b3a31fa4470808389db27a6"
server_key_version: "encryptionkey@0"
I20260929 15:44:44.663504 4341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:44.691310 4341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:44.697695 4341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:44.698081 4341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:44.699694 4341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:44.701673 4341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:44.701766 4341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:44.701915 4341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:44.701982 4341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:44.743883 4341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45519
I20260929 15:44:44.743930 4468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45519 every 8 connection(s)
I20260929 15:44:44.745795 4341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:44.746724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4341
I20260929 15:44:44.747016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:44.747857 388 external_mini_cluster.cc:1428] Setting key 312a082ef69fb42c8bbf81766b10e274
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:44.772279 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.751325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54799 (local address 127.0.97.62:45861)
0929 15:44:44.756082 (+ 4757us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:44.756087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:44.756106 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:44:44.756160 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:44.756222 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:44.756308 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:44.756419 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:44.758190 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.758999 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.761262 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.761684 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.761854 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.765619 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:44.765625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:44.765636 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:44.765680 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:44.768854 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.769699 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.769711 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.769714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.769834 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.770292 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.770296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.770299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.770571 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:44.770731 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:44.771588 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 15:44:44.771851 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4651,"thread_start_us":4548,"threads_started":1}
I20260929 15:44:44.773321 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.750705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45861 (local address 127.0.97.3:54799)
0929 15:44:44.752232 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:44.752290 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:44.755799 (+ 3509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:44.756655 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:44.756664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:44.757341 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:44.757973 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:44.757993 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.759217 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.759222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:44.761086 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:44.761101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.761960 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.761969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:44.762244 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.763028 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:44.763052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:44.765380 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:44.769024 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:44.769034 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:44.769059 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:44.769535 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:44.769951 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:44.769956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:44.769960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:44.770152 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:44.770711 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:44.770719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:44.771155 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 15:44:44.771481 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 15:44:44.772074 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":938,"threads_started":1}
I20260929 15:44:44.775194 4469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45861
I20260929 15:44:44.775720 4469 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:44.776921 4469 heartbeater.cc:511] Master 127.0.97.62:45861 requested a full tablet report, sending...
I20260929 15:44:44.778813 4012 ts_manager.cc:194] Registered new tserver with Master: a94d9ff8664646068282972b2a60e89e (127.0.97.3:45519)
I20260929 15:44:44.779752 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54799
I20260929 15:44:44.790616 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:44.853964 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.833465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.0.97.1:34107)
0929 15:44:44.834167 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:44.834175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:44.834203 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:44:44.834266 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:44.834272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:44.834492 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:44.834631 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:44.836113 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.837332 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.838871 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.839652 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.839824 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.845074 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:44.845091 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:44.845108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:44.845158 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:44.849586 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.850779 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.850790 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.850792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.851033 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.851700 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.851704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.851707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.852144 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:44.852567 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:44.853206 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 15:44:44.853550 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":383,"threads_started":1}
Sep 29 15:44:44 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:44.878173 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.860386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.97.62:45861)
0929 15:44:44.860627 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:44.860635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:44.860826 (+ 191us) server_negotiation.cc:407] Connection header received
0929 15:44:44.860958 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:44.860962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:44.861066 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:44.861163 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:44.862617 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.863534 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.865166 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.865671 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.865799 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.871018 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:44.871024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:44.871027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:44.871076 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:44.874086 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.875250 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.875255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.875258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.875439 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.876004 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.876009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.876011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.876495 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:44.876709 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:44.877423 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 15:44:44.877771 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 15:44:44.879076 4012 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010
I20260929 15:44:44.880659 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4073
I20260929 15:44:44.900043 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:44.882057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.0.97.1:34107)
0929 15:44:44.882406 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:44.882415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:44.882653 (+ 238us) server_negotiation.cc:407] Connection header received
0929 15:44:44.882908 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:44.882913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:44.882988 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:44.883117 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:44.884812 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.885587 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.887698 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:44.888272 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:44.888469 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:44.891906 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:44.891913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:44.891917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:44.891975 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:44.896270 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.897272 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.897277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.897279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.897461 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:44.897979 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:44.897985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:44.897988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:44.898388 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:44.898722 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:44.899359 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 15:44:44.899620 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260929 15:44:44.900972 4196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:44.993885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4073
I20260929 15:44:45.013098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4207
Sep 29 15:44:45 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:45.034673 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:45.014377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.0.97.2:43573)
0929 15:44:45.015001 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:45.015009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:45.015071 (+ 62us) server_negotiation.cc:407] Connection header received
0929 15:44:45.015169 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:45.015176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:45.015414 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:45.015581 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:45.017285 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:45.018327 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:45.020382 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:45.021039 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:45.021196 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:45.026639 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:45.026658 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:45.026676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:45.026724 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:45.030834 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:45.031979 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:45.032001 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:45.032004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:45.032281 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:45.032801 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:45.032807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:45.032811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:45.033220 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:45.033582 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:45.033949 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 15:44:45.034253 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":319,"threads_started":1}
I20260929 15:44:45.036728 4330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:45.127210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4207
I20260929 15:44:45.144471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4341
Sep 29 15:44:45 dist-test-slave-7491 krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:45.165838 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:45.145585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.0.97.3:45519)
0929 15:44:45.146190 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:45.146197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:45.146229 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:44:45.146293 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:45.146300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:45.146535 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:45.146740 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:45.148123 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:45.149305 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:45.150782 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:45.151589 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:45.151907 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:45.158158 (+ 6251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:45.158177 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:45.158196 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:45.158257 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:45.161763 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:45.162808 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:45.162819 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:45.162822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:45.163042 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:45.163599 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:45.163604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:45.163606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:45.164120 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:45.164613 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:45.165222 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 15:44:45.165505 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":306,"threads_started":1}
I20260929 15:44:45.167311 4464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:45.263635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4341
I20260929 15:44:45.281491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 3980
I20260929 15:44:45.282991 4042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:45.404552 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 3980
2026-09-29T15:44:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4532 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:45 dist-test-slave-7491 krb5kdc[4489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:45 dist-test-slave-7491 krb5kdc[4489](info): set up 2 sockets
Sep 29 15:44:45 dist-test-slave-7491 krb5kdc[4489](info): commencing operation
krb5kdc: starting...
W20260929 15:44:46.570165 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:46 dist-test-slave-7491 krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:46.822175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44487
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:47.219322 4502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:47.219672 4502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:47.219766 4502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:47.227605 4502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:47.227756 4502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:47.227807 4502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:47.227859 4502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:47.227900 4502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:47.240144 4502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:47.242321 4502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:47.243902 4502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:47.259591 4508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:47.259642 4510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:47.259665 4507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:47.261281 4502 server_base.cc:1034] running on GCE node
I20260929 15:44:47.262367 4502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:47.264472 4502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:47.265754 4502 hybrid_clock.cc:648] HybridClock initialized: now 1790696687265704 us; error 77 us; skew 500 ppm
I20260929 15:44:47.266207 4502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:47 dist-test-slave-7491 krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:47.270784 4502 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:44:47.272498 4502 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:47.272661 4502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:47.273056 4502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:47.276629 4502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "478ca1b09ed141abb1798f581abe1b60"
format_stamp: "Formatted at 2026-09-29 15:44:47 on dist-test-slave-7491"
server_key: "b2319bad537b1042661e2f2c715ec352"
server_key_iv: "9480964b01e68bb7f2b328ab733d30d8"
server_key_version: "encryptionkey@0"
I20260929 15:44:47.277622 4502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "478ca1b09ed141abb1798f581abe1b60"
format_stamp: "Formatted at 2026-09-29 15:44:47 on dist-test-slave-7491"
server_key: "b2319bad537b1042661e2f2c715ec352"
server_key_iv: "9480964b01e68bb7f2b328ab733d30d8"
server_key_version: "encryptionkey@0"
I20260929 15:44:47.284595 4502 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 15:44:47.289044 4516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:47.290781 4502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:44:47.291039 4502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "478ca1b09ed141abb1798f581abe1b60"
format_stamp: "Formatted at 2026-09-29 15:44:47 on dist-test-slave-7491"
server_key: "b2319bad537b1042661e2f2c715ec352"
server_key_iv: "9480964b01e68bb7f2b328ab733d30d8"
server_key_version: "encryptionkey@0"
I20260929 15:44:47.291322 4502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:47.309057 4502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:47.315111 4502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:47.315486 4502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:47.341058 4502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44487
I20260929 15:44:47.341106 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44487 every 8 connection(s)
I20260929 15:44:47.342370 4502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:47.347038 4568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:47.349081 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4502
I20260929 15:44:47.349313 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:47.350008 388 external_mini_cluster.cc:1428] Setting key 981bb18779513a684c3405065b74e978
I20260929 15:44:47.365023 4568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60: Bootstrap starting.
Sep 29 15:44:47 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:47.368923 4568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:47.370416 4568 log.cc:824] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:47.375056 4568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60: No bootstrap required, opened a new log
I20260929 15:44:47.380772 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:47.352853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.0.97.62:44487)
0929 15:44:47.354323 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:47.354342 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:47.354419 (+ 77us) server_negotiation.cc:407] Connection header received
0929 15:44:47.356612 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:47.356648 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:47.357267 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:47.357948 (+ 681us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:47.359991 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:47.361537 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:47.362963 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:47.363797 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:47.364034 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:47.368742 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:47.368751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:47.368771 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:47.368823 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:47.373396 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:47.374662 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:47.374680 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:47.374689 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:47.374902 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:47.375405 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:47.375411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:47.375414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:47.376508 (+ 1094us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:47.376878 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:47.378001 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0929 15:44:47.378882 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":692,"threads_started":1}
I20260929 15:44:47.386631 4568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478ca1b09ed141abb1798f581abe1b60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44487 } }
I20260929 15:44:47.387082 4568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:47.387154 4568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478ca1b09ed141abb1798f581abe1b60, State: Initialized, Role: FOLLOWER
I20260929 15:44:47.387764 4568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478ca1b09ed141abb1798f581abe1b60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44487 } }
I20260929 15:44:47.388000 4568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:47.388106 4568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:47.388356 4568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:47.391323 4568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478ca1b09ed141abb1798f581abe1b60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44487 } }
I20260929 15:44:47.392526 4568 leader_election.cc:302] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 478ca1b09ed141abb1798f581abe1b60; no voters:
I20260929 15:44:47.393347 4568 leader_election.cc:288] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:47.393576 4573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:47.394516 4573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [term 1 LEADER]: Becoming Leader. State: Replica: 478ca1b09ed141abb1798f581abe1b60, State: Running, Role: LEADER
I20260929 15:44:47.395548 4573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478ca1b09ed141abb1798f581abe1b60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44487 } }
I20260929 15:44:47.397193 4568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:47.401007 4575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 478ca1b09ed141abb1798f581abe1b60. Latest consensus state: current_term: 1 leader_uuid: "478ca1b09ed141abb1798f581abe1b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478ca1b09ed141abb1798f581abe1b60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44487 } } }
I20260929 15:44:47.400683 4574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "478ca1b09ed141abb1798f581abe1b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "478ca1b09ed141abb1798f581abe1b60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44487 } } }
I20260929 15:44:47.401749 4575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:47.401757 4574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:47.403427 4579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:47.410269 4579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:47.422417 4579 catalog_manager.cc:1347] Generated new cluster ID: f1d02c22cd8140429fab6092530c9c52
I20260929 15:44:47.422596 4579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:47.454496 4579 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:47.457037 4579 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:47.468894 4579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60: Generated new TSK 0
I20260929 15:44:47.469990 4579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:44:47.528788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44487
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:47.965090 4594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:47.965487 4594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:47.965668 4594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:47.974087 4594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:47.974278 4594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:47.974475 4594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:47.987711 4594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:47.989388 4594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:47.990882 4594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:48.008847 4600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:48.008922 4602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:48.008909 4599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:48.010519 4594 server_base.cc:1034] running on GCE node
I20260929 15:44:48.011236 4594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:48.012436 4594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:48.013787 4594 hybrid_clock.cc:648] HybridClock initialized: now 1790696688013698 us; error 48 us; skew 500 ppm
I20260929 15:44:48.014411 4594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:48 dist-test-slave-7491 krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:48.019366 4594 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:44:48.021193 4594 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:48.021323 4594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:48.021636 4594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:48.025455 4594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "8bcfd41893ac495e9d6851f3c3697cb2"
format_stamp: "Formatted at 2026-09-29 15:44:48 on dist-test-slave-7491"
server_key: "ad80e995e1de00af19bdfa12ee41cf60"
server_key_iv: "c3d3fab5606b47467fa7e5b03e2e384d"
server_key_version: "encryptionkey@0"
I20260929 15:44:48.026491 4594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8bcfd41893ac495e9d6851f3c3697cb2"
format_stamp: "Formatted at 2026-09-29 15:44:48 on dist-test-slave-7491"
server_key: "ad80e995e1de00af19bdfa12ee41cf60"
server_key_iv: "c3d3fab5606b47467fa7e5b03e2e384d"
server_key_version: "encryptionkey@0"
I20260929 15:44:48.033104 4594 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:44:48.037678 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:48.040025 4594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 15:44:48.040372 4594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "8bcfd41893ac495e9d6851f3c3697cb2"
format_stamp: "Formatted at 2026-09-29 15:44:48 on dist-test-slave-7491"
server_key: "ad80e995e1de00af19bdfa12ee41cf60"
server_key_iv: "c3d3fab5606b47467fa7e5b03e2e384d"
server_key_version: "encryptionkey@0"
I20260929 15:44:48.040758 4594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:48.067364 4594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:48.073076 4594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:48.073460 4594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:48.074676 4594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:48.076556 4594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:48.076725 4594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:48.076881 4594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:48.076987 4594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:48.119879 4594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44047
I20260929 15:44:48.119951 4720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44047 every 8 connection(s)
I20260929 15:44:48.120918 4594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:48.129783 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4594
I20260929 15:44:48.129956 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:48.130640 388 external_mini_cluster.cc:1428] Setting key 87aac3bfcbf42a853397d038c46be54a
Sep 29 15:44:48 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:48.146873 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:48.124322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48749 (local address 127.0.97.62:44487)
0929 15:44:48.125001 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:48.125008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:48.126332 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 15:44:48.128695 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:48.128701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:48.128909 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:48.129078 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:48.131565 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.132492 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.134379 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.134818 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.134975 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:48.139801 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:48.139808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:48.139813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:48.139865 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:48.142905 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:48.143927 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:48.143933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:48.143935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:48.144071 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:48.144602 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:48.144610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:48.144616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:48.144956 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:48.145147 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:48.146111 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 15:44:48.146391 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":257,"threads_started":1}
I20260929 15:44:48.147960 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:48.125176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44487 (local address 127.0.97.1:48749)
0929 15:44:48.126114 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:48.126170 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:48.128249 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:48.129463 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:48.129475 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:48.130419 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:48.131315 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:48.131337 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.132711 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.132715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:48.134185 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:48.134199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.135100 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.135106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:48.135477 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:48.136445 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:48.136473 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:48.139427 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:48.143148 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:48.143160 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:48.143187 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:48.143742 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:48.144224 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:48.144233 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:48.144236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:48.144469 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:48.145106 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:48.145116 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:48.145625 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 15:44:48.145986 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 15:44:48.146644 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":413,"threads_started":1}
I20260929 15:44:48.149896 4721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44487
I20260929 15:44:48.150290 4721 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:48.151242 4721 heartbeater.cc:511] Master 127.0.97.62:44487 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:44:48.154598 4533 ts_manager.cc:194] Registered new tserver with Master: 8bcfd41893ac495e9d6851f3c3697cb2 (127.0.97.1:44047)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:44:48.156901 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:44:48.184374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44487
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:48.604907 4727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:48.605381 4727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:48.605566 4727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:48.614609 4727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:48.614807 4727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:48.614961 4727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:48.627415 4727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:48.628902 4727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:48.630352 4727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:48.648276 4732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:48.648300 4733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:48.648221 4735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:48.649474 4727 server_base.cc:1034] running on GCE node
I20260929 15:44:48.650421 4727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:48.651806 4727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:48.655277 4727 hybrid_clock.cc:648] HybridClock initialized: now 1790696688655157 us; error 95 us; skew 500 ppm
I20260929 15:44:48.655776 4727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:48 dist-test-slave-7491 krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:48.660372 4727 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:44:48.662091 4727 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:48.662305 4727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:48.662686 4727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:48.666177 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "a7a2ece4253b4edb885e71cecb0b6dad"
format_stamp: "Formatted at 2026-09-29 15:44:48 on dist-test-slave-7491"
server_key: "46c2b71cd0f2a44b554dd728ace1dca7"
server_key_iv: "d9b6a379a0ae34bf02c6d6692370cac8"
server_key_version: "encryptionkey@0"
I20260929 15:44:48.667168 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a7a2ece4253b4edb885e71cecb0b6dad"
format_stamp: "Formatted at 2026-09-29 15:44:48 on dist-test-slave-7491"
server_key: "46c2b71cd0f2a44b554dd728ace1dca7"
server_key_iv: "d9b6a379a0ae34bf02c6d6692370cac8"
server_key_version: "encryptionkey@0"
I20260929 15:44:48.674237 4727 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20260929 15:44:48.679381 4741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:48.681712 4727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:48.682049 4727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "a7a2ece4253b4edb885e71cecb0b6dad"
format_stamp: "Formatted at 2026-09-29 15:44:48 on dist-test-slave-7491"
server_key: "46c2b71cd0f2a44b554dd728ace1dca7"
server_key_iv: "d9b6a379a0ae34bf02c6d6692370cac8"
server_key_version: "encryptionkey@0"
I20260929 15:44:48.682360 4727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:48.718068 4727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:48.724046 4727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:48.724424 4727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:48.725634 4727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:48.727375 4727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:48.727526 4727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:48.727676 4727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:48.727767 4727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:48.769950 4727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43757
I20260929 15:44:48.769977 4853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43757 every 8 connection(s)
I20260929 15:44:48.771095 4727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:48.772749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4727
I20260929 15:44:48.772941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:48.773429 388 external_mini_cluster.cc:1428] Setting key 6ce89d36fad88e617f67fd0286cbf68d
Sep 29 15:44:48 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:44:48.803102 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:48.774377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35221 (local address 127.0.97.62:44487)
0929 15:44:48.781442 (+ 7065us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:48.781449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:48.781476 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:44:48.781552 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:48.781560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:48.781664 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:48.781802 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:48.783784 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.785087 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.787139 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.788105 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.788318 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:48.793121 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:48.793132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:48.793139 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:48.793217 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:48.797991 (+ 4774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:48.799394 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:48.799403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:48.799407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:48.799637 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:48.800625 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:48.800630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:48.800633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:48.801004 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:48.801152 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:48.802557 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0929 15:44:48.802825 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6879,"thread_start_us":6725,"threads_started":1}
I20260929 15:44:48.805603 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:48.774894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44487 (local address 127.0.97.2:35221)
0929 15:44:48.776088 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:48.776160 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:48.780585 (+ 4425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:48.782114 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:48.782125 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:48.782761 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:48.783563 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:48.783585 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.785295 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.785300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:48.786931 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:48.786952 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:48.788415 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:48.788420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:48.788663 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:48.789532 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:48.789585 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:48.792613 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:48.798280 (+ 5667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:48.798291 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:48.798309 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:48.799171 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:48.799960 (+ 789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:48.799966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:48.799971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:48.800429 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:48.801280 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:48.801294 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:48.801928 (+ 634us) client_negotiation.cc:769] Sending connection context
0929 15:44:48.802372 (+ 444us) client_negotiation.cc:240] Negotiation successful
0929 15:44:48.803511 (+ 1139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":485,"threads_started":1}
I20260929 15:44:48.807502 4854 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44487
I20260929 15:44:48.807931 4854 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:48.808928 4854 heartbeater.cc:511] Master 127.0.97.62:44487 requested a full tablet report, sending...
I20260929 15:44:48.810964 4533 ts_manager.cc:194] Registered new tserver with Master: a7a2ece4253b4edb885e71cecb0b6dad (127.0.97.2:43757)
I20260929 15:44:48.812072 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:44:48.832842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44487
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:44:49.161657 4721 heartbeater.cc:503] Master 127.0.97.62:44487 was elected leader, sending a full tablet report...
W20260929 15:44:49.223100 4860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:49.223619 4860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:49.223839 4860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:49.233882 4860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:49.234127 4860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:49.234333 4860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:49.247381 4860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:49.249085 4860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:49.250613 4860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:49.266558 4865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:49.266537 4866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:49.267020 4868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:49.267831 4860 server_base.cc:1034] running on GCE node
I20260929 15:44:49.268671 4860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:49.269896 4860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:49.271214 4860 hybrid_clock.cc:648] HybridClock initialized: now 1790696689271149 us; error 60 us; skew 500 ppm
I20260929 15:44:49.271777 4860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:44:49 dist-test-slave-7491 krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:44:49.281833 4860 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:44:49.283668 4860 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:49.283861 4860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:49.284332 4860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:49.288065 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "73975a080ab14740bb02eea01d24581e"
format_stamp: "Formatted at 2026-09-29 15:44:49 on dist-test-slave-7491"
server_key: "ff01924cba25fed325a9e8755d13223d"
server_key_iv: "7b978387d368d481392fce3e88e31782"
server_key_version: "encryptionkey@0"
I20260929 15:44:49.289143 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "73975a080ab14740bb02eea01d24581e"
format_stamp: "Formatted at 2026-09-29 15:44:49 on dist-test-slave-7491"
server_key: "ff01924cba25fed325a9e8755d13223d"
server_key_iv: "7b978387d368d481392fce3e88e31782"
server_key_version: "encryptionkey@0"
I20260929 15:44:49.295106 4860 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:49.300132 4874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:49.302233 4860 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:49.302574 4860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "73975a080ab14740bb02eea01d24581e"
format_stamp: "Formatted at 2026-09-29 15:44:49 on dist-test-slave-7491"
server_key: "ff01924cba25fed325a9e8755d13223d"
server_key_iv: "7b978387d368d481392fce3e88e31782"
server_key_version: "encryptionkey@0"
I20260929 15:44:49.302917 4860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:49.323805 4860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:49.330842 4860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:49.331248 4860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:49.332674 4860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:49.334784 4860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:49.334928 4860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:49.335112 4860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:49.335216 4860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:49.376878 4860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42811
I20260929 15:44:49.376924 4986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42811 every 8 connection(s)
I20260929 15:44:49.377857 4860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:49.379519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 4860
I20260929 15:44:49.379750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:49.380237 388 external_mini_cluster.cc:1428] Setting key d52bb866900fd4f90f83c25f77390817
Sep 29 15:44:49 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:44:49.404799 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.381874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43483 (local address 127.0.97.62:44487)
0929 15:44:49.382559 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.382566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.384324 (+ 1758us) server_negotiation.cc:407] Connection header received
0929 15:44:49.386562 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.386567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.386714 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.386838 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:49.388982 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.389993 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.391968 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.392610 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.392765 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.397152 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:49.397158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:49.397162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:49.397201 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:49.400842 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.401831 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.401837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.401841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.401995 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.402495 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.402500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.402506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.402805 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:49.403002 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.403978 (+ 976us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.404336 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
I20260929 15:44:49.405892 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.382699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44487 (local address 127.0.97.3:43483)
0929 15:44:49.383837 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:49.384067 (+ 230us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:49.386204 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:49.387165 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:49.387176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:49.387864 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:49.388674 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.388698 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.390255 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.390259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.391652 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.391670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.392955 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.392962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.393197 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.394064 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:44:49.394097 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:49.396846 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:49.401073 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:49.401086 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:49.401116 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:49.401659 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:49.402129 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:44:49.402135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:44:49.402139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:44:49.402352 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:44:49.402976 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:49.402985 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:49.403488 (+ 503us) client_negotiation.cc:769] Sending connection context
0929 15:44:49.403863 (+ 375us) client_negotiation.cc:240] Negotiation successful
0929 15:44:49.404567 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":524,"threads_started":1}
I20260929 15:44:49.408336 4987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44487
I20260929 15:44:49.408677 4987 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:49.409524 4987 heartbeater.cc:511] Master 127.0.97.62:44487 requested a full tablet report, sending...
I20260929 15:44:49.411675 4533 ts_manager.cc:194] Registered new tserver with Master: 73975a080ab14740bb02eea01d24581e (127.0.97.3:42811)
I20260929 15:44:49.412626 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:44:49.413033 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43483
I20260929 15:44:49.434834 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.418899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.0.97.62:44487)
0929 15:44:49.419247 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.419254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.419575 (+ 321us) server_negotiation.cc:407] Connection header received
0929 15:44:49.420064 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.420068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.420145 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.420311 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:49.422240 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.423035 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.424571 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.425029 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.425149 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.427819 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:49.427829 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:49.427833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:49.427874 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:49.431358 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.432310 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.432315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.432318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.432467 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.432877 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.432890 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.432892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.433241 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:49.433616 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.434035 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.434387 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 15:44:49.441331 4533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:49.450377 4533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:49.480046 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.467915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43757 (local address 127.0.0.1:60104)
0929 15:44:49.468804 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:49.468855 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:49.469083 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:49.471835 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:49.471848 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:49.471911 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:49.472571 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.472609 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.475074 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.475080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.476939 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.476954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.478904 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.478918 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.479170 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.479240 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 15:44:49.479371 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 15:44:49.479593 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":405,"threads_started":1}
I20260929 15:44:49.480510 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.468202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.0.97.2:43757)
0929 15:44:49.470994 (+ 2792us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.471005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.471037 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:44:49.471144 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.471151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.471487 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.472406 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:49.472900 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.474891 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.477128 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.478613 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.479471 (+ 858us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.479692 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.479789 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.479989 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2612,"thread_start_us":2473,"threads_started":1}
I20260929 15:44:49.483136 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.470374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.0.97.3:42811)
0929 15:44:49.471463 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.471475 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.472707 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 15:44:49.472874 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.472886 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.473242 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.473498 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:49.475419 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.477372 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.480215 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.481975 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.482197 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.482455 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.482629 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.482799 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":696,"threads_started":1}
I20260929 15:44:49.483166 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.469549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42811 (local address 127.0.0.1:47998)
0929 15:44:49.472127 (+ 2578us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:49.472158 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:49.472446 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:49.473535 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:49.473540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:49.473579 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:49.474616 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.474633 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.477584 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.477590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.479943 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.479958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.482230 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.482237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.482425 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.482463 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:44:49.482570 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 15:44:49.482721 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2246,"mutex_wait_us":2,"thread_start_us":2087,"threads_started":1}
I20260929 15:44:49.485176 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.468754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44047 (local address 127.0.0.1:47522)
0929 15:44:49.469788 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:49.469808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:49.470454 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:49.473973 (+ 3519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:49.473979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:49.474008 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:49.475076 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.475087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.477658 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.477667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.480920 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.480933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.483648 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.483657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.484426 (+ 769us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.484465 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:44:49.484593 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:44:49.484749 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":637,"spinlock_wait_cycles":1408,"thread_start_us":885,"threads_started":1}
I20260929 15:44:49.485744 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.469402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.0.97.1:44047)
0929 15:44:49.473338 (+ 3936us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.473349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.473384 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:44:49.473499 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.473508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.473740 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.473984 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:49.475305 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.477424 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.481137 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.482978 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.483319 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.483651 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.484860 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.485105 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3724,"thread_start_us":3479,"threads_started":1}
I20260929 15:44:49.485993 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 14338b461eb8494d81353036f63d8a40 (DEFAULT_TABLE table=test-table [id=4a51baa0e2a5403486f05b5e71a1fc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:49.488626 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14338b461eb8494d81353036f63d8a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:49.487984 4922 tablet_service.cc:1467] Processing CreateTablet for tablet 14338b461eb8494d81353036f63d8a40 (DEFAULT_TABLE table=test-table [id=4a51baa0e2a5403486f05b5e71a1fc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:49.489584 4922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14338b461eb8494d81353036f63d8a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:49.490034 4656 tablet_service.cc:1467] Processing CreateTablet for tablet 14338b461eb8494d81353036f63d8a40 (DEFAULT_TABLE table=test-table [id=4a51baa0e2a5403486f05b5e71a1fc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:49.491501 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14338b461eb8494d81353036f63d8a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:49.502978 5006 tablet_bootstrap.cc:492] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Bootstrap starting.
I20260929 15:44:49.506330 5007 tablet_bootstrap.cc:492] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Bootstrap starting.
I20260929 15:44:49.508074 5008 tablet_bootstrap.cc:492] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Bootstrap starting.
I20260929 15:44:49.509989 5006 tablet_bootstrap.cc:654] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:49.511626 5008 tablet_bootstrap.cc:654] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:49.511825 5006 log.cc:824] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:49.512156 5007 tablet_bootstrap.cc:654] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:49.512912 5008 log.cc:824] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:49.514443 5007 log.cc:824] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:49.516563 5006 tablet_bootstrap.cc:492] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: No bootstrap required, opened a new log
I20260929 15:44:49.516995 5006 ts_tablet_manager.cc:1397] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260929 15:44:49.517274 5008 tablet_bootstrap.cc:492] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: No bootstrap required, opened a new log
I20260929 15:44:49.517604 5008 ts_tablet_manager.cc:1397] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 15:44:49.517808 5007 tablet_bootstrap.cc:492] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: No bootstrap required, opened a new log
I20260929 15:44:49.518136 5007 ts_tablet_manager.cc:1397] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260929 15:44:49.526486 5007 raft_consensus.cc:348] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.526979 5007 raft_consensus.cc:374] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:49.527076 5007 raft_consensus.cc:729] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73975a080ab14740bb02eea01d24581e, State: Initialized, Role: FOLLOWER
I20260929 15:44:49.527801 5007 consensus_queue.cc:260] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.529122 4987 heartbeater.cc:503] Master 127.0.97.62:44487 was elected leader, sending a full tablet report...
I20260929 15:44:49.529891 5007 ts_tablet_manager.cc:1428] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 15:44:49.529655 5008 raft_consensus.cc:348] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.529655 5006 raft_consensus.cc:348] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.530128 5006 raft_consensus.cc:374] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:49.530128 5008 raft_consensus.cc:374] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:49.530225 5006 raft_consensus.cc:729] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a2ece4253b4edb885e71cecb0b6dad, State: Initialized, Role: FOLLOWER
I20260929 15:44:49.530228 5008 raft_consensus.cc:729] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcfd41893ac495e9d6851f3c3697cb2, State: Initialized, Role: FOLLOWER
I20260929 15:44:49.530894 5008 consensus_queue.cc:260] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.530895 5006 consensus_queue.cc:260] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.532603 4854 heartbeater.cc:503] Master 127.0.97.62:44487 was elected leader, sending a full tablet report...
I20260929 15:44:49.532994 5008 ts_tablet_manager.cc:1428] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 15:44:49.533177 5006 ts_tablet_manager.cc:1428] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260929 15:44:49.624094 4722 tablet.cc:2369] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:44:49.629897 4988 tablet.cc:2369] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:49.670643 5012 raft_consensus.cc:482] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:49.671005 5012 raft_consensus.cc:504] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.673277 5012 leader_election.cc:288] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7a2ece4253b4edb885e71cecb0b6dad (127.0.97.2:43757), 8bcfd41893ac495e9d6851f3c3697cb2 (127.0.97.1:44047)
I20260929 15:44:49.680044 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.673785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54233 (local address 127.0.97.2:43757)
0929 15:44:49.674034 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.674043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.674074 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:44:49.674300 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.674304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.674364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.674475 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:49.674862 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.675594 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.678042 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.678899 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.679091 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.679205 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.679554 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.679714 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 15:44:49.680070 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.673653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43757 (local address 127.0.97.3:54233)
0929 15:44:49.673934 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:49.673957 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:49.674099 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:49.674502 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:49.674506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:49.674534 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:49.674721 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.674729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.675749 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.675754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.677730 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.677757 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.679132 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.679139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.679362 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.679392 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:44:49.679524 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 15:44:49.679705 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 15:44:49.680214 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.674004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48359 (local address 127.0.97.1:44047)
0929 15:44:49.674222 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.674228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.674697 (+ 469us) server_negotiation.cc:407] Connection header received
0929 15:44:49.674911 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.674915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.674972 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.675088 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:44:49.675598 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.676455 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.678300 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.679094 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.679276 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.679394 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.679752 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.679886 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:44:49.680516 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.673742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44047 (local address 127.0.97.3:48359)
0929 15:44:49.674583 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:49.674599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:49.674765 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:49.675103 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:49.675107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:49.675129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:44:49.675437 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.675446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.676619 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.676627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.678102 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:49.678118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.679363 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.679372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:49.679570 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.679601 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:44:49.679724 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 15:44:49.679887 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":5,"thread_start_us":530,"threads_started":1}
I20260929 15:44:49.681767 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14338b461eb8494d81353036f63d8a40" candidate_uuid: "73975a080ab14740bb02eea01d24581e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" is_pre_election: true
I20260929 15:44:49.681751 4676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14338b461eb8494d81353036f63d8a40" candidate_uuid: "73975a080ab14740bb02eea01d24581e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" is_pre_election: true
I20260929 15:44:49.682548 4809 raft_consensus.cc:2454] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73975a080ab14740bb02eea01d24581e in term 0.
I20260929 15:44:49.682548 4676 raft_consensus.cc:2454] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73975a080ab14740bb02eea01d24581e in term 0.
I20260929 15:44:49.683434 4877 leader_election.cc:302] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73975a080ab14740bb02eea01d24581e, a7a2ece4253b4edb885e71cecb0b6dad; no voters:
I20260929 15:44:49.683845 5012 raft_consensus.cc:2781] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:49.683998 5012 raft_consensus.cc:482] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:49.684113 5012 raft_consensus.cc:3037] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:49.686678 5012 raft_consensus.cc:504] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.687474 5012 leader_election.cc:288] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [CANDIDATE]: Term 1 election: Requested vote from peers a7a2ece4253b4edb885e71cecb0b6dad (127.0.97.2:43757), 8bcfd41893ac495e9d6851f3c3697cb2 (127.0.97.1:44047)
I20260929 15:44:49.688011 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14338b461eb8494d81353036f63d8a40" candidate_uuid: "73975a080ab14740bb02eea01d24581e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a2ece4253b4edb885e71cecb0b6dad"
I20260929 15:44:49.688115 4676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14338b461eb8494d81353036f63d8a40" candidate_uuid: "73975a080ab14740bb02eea01d24581e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bcfd41893ac495e9d6851f3c3697cb2"
I20260929 15:44:49.688283 4809 raft_consensus.cc:3037] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:49.688437 4676 raft_consensus.cc:3037] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:49.690750 4809 raft_consensus.cc:2454] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73975a080ab14740bb02eea01d24581e in term 1.
I20260929 15:44:49.690752 4676 raft_consensus.cc:2454] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73975a080ab14740bb02eea01d24581e in term 1.
I20260929 15:44:49.691313 4877 leader_election.cc:302] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73975a080ab14740bb02eea01d24581e, a7a2ece4253b4edb885e71cecb0b6dad; no voters:
I20260929 15:44:49.691684 5012 raft_consensus.cc:2781] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:49.692548 5012 raft_consensus.cc:686] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 1 LEADER]: Becoming Leader. State: Replica: 73975a080ab14740bb02eea01d24581e, State: Running, Role: LEADER
I20260929 15:44:49.693197 5012 consensus_queue.cc:237] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } }
I20260929 15:44:49.698519 4532 catalog_manager.cc:5495] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e reported cstate change: term changed from 0 to 1, leader changed from <none> to 73975a080ab14740bb02eea01d24581e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "73975a080ab14740bb02eea01d24581e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73975a080ab14740bb02eea01d24581e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 } health_report { overall_health: UNKNOWN } } }
I20260929 15:44:49.726138 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.719647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.0.97.3:42811)
0929 15:44:49.719867 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.719872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.719894 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:44:49.720019 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.720023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.720148 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.720395 (+ 247us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:44:49.720974 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.721767 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.723342 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.723817 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.723953 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.724700 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:44:49.725289 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:44:49.725427 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.725713 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.725797 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:44:49.735759 4676 raft_consensus.cc:1261] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 1 FOLLOWER]: Refusing update from remote peer 73975a080ab14740bb02eea01d24581e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:49.735755 4809 raft_consensus.cc:1261] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 1 FOLLOWER]: Refusing update from remote peer 73975a080ab14740bb02eea01d24581e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:49.737057 5016 consensus_queue.cc:1035] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a2ece4253b4edb885e71cecb0b6dad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:49.737452 5012 consensus_queue.cc:1035] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bcfd41893ac495e9d6851f3c3697cb2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:49.778702 4533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:49.779125 4533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:49.785447 4533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 478ca1b09ed141abb1798f581abe1b60: Sending DeleteTablet for 3 replicas of tablet 14338b461eb8494d81353036f63d8a40
I20260929 15:44:49.786957 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 14338b461eb8494d81353036f63d8a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60104
I20260929 15:44:49.787772 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet 14338b461eb8494d81353036f63d8a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47522
I20260929 15:44:49.789076 4922 tablet_service.cc:1514] Processing DeleteTablet for tablet 14338b461eb8494d81353036f63d8a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47998
I20260929 15:44:49.789268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4594
I20260929 15:44:49.797205 5031 tablet_replica.cc:330] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: stopping tablet replica
I20260929 15:44:49.797842 5031 raft_consensus.cc:2229] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:49.798529 5031 raft_consensus.cc:2258] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:49.801620 5031 ts_tablet_manager.cc:1905] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:44:49 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:44:49.811698 5031 ts_tablet_manager.cc:1918] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:49.811934 5031 log.cc:1196] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/14338b461eb8494d81353036f63d8a40
I20260929 15:44:49.812676 5031 ts_tablet_manager.cc:1939] T 14338b461eb8494d81353036f63d8a40 P 8bcfd41893ac495e9d6851f3c3697cb2: Deleting consensus metadata
I20260929 15:44:49.815086 4520 catalog_manager.cc:4841] TS 8bcfd41893ac495e9d6851f3c3697cb2 (127.0.97.1:44047): tablet 14338b461eb8494d81353036f63d8a40 (table test-table [id=4a51baa0e2a5403486f05b5e71a1fc12]) successfully deleted
I20260929 15:44:49.815802 5030 tablet_replica.cc:330] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: stopping tablet replica
I20260929 15:44:49.816082 5029 tablet_replica.cc:330] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: stopping tablet replica
I20260929 15:44:49.816684 5029 raft_consensus.cc:2229] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:49.816753 5030 raft_consensus.cc:2229] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:49.817277 5029 raft_consensus.cc:2258] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:49.818280 5030 raft_consensus.cc:2258] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:49.819471 5029 ts_tablet_manager.cc:1905] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:49.819964 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.790690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.0.97.1:44047)
0929 15:44:49.790873 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.790879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.796804 (+ 5925us) server_negotiation.cc:407] Connection header received
0929 15:44:49.797137 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.797145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.797267 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.797351 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:49.799375 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.800527 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.802557 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.803245 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.803350 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.810491 (+ 7141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:49.810520 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:49.810548 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:49.810624 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:49.815398 (+ 4774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.816576 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.816585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.816588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.816786 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.817305 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.817310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.817314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.818338 (+ 1024us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:49.818724 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.819354 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.819604 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:44:49.821595 4716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:49.821635 5030 ts_tablet_manager.cc:1905] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:49.828323 5029 ts_tablet_manager.cc:1918] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:49.828619 5029 log.cc:1196] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/14338b461eb8494d81353036f63d8a40
I20260929 15:44:49.829344 5029 ts_tablet_manager.cc:1939] T 14338b461eb8494d81353036f63d8a40 P a7a2ece4253b4edb885e71cecb0b6dad: Deleting consensus metadata
I20260929 15:44:49.832571 4519 catalog_manager.cc:4841] TS a7a2ece4253b4edb885e71cecb0b6dad (127.0.97.2:43757): tablet 14338b461eb8494d81353036f63d8a40 (table test-table [id=4a51baa0e2a5403486f05b5e71a1fc12]) successfully deleted
I20260929 15:44:49.834877 5030 ts_tablet_manager.cc:1918] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:49.835466 5030 log.cc:1196] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/14338b461eb8494d81353036f63d8a40
I20260929 15:44:49.836395 5030 ts_tablet_manager.cc:1939] T 14338b461eb8494d81353036f63d8a40 P 73975a080ab14740bb02eea01d24581e: Deleting consensus metadata
I20260929 15:44:49.838315 4519 catalog_manager.cc:4841] TS 73975a080ab14740bb02eea01d24581e (127.0.97.3:42811): tablet 14338b461eb8494d81353036f63d8a40 (table test-table [id=4a51baa0e2a5403486f05b5e71a1fc12]) successfully deleted
I20260929 15:44:49.957571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4594
I20260929 15:44:49.978271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4727
Sep 29 15:44:49 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:44:49.996729 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:49.979164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.0.97.2:43757)
0929 15:44:49.979403 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:49.979409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:49.979431 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:44:49.979553 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:49.979558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:49.979631 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:49.979743 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:49.981090 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.981858 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.983181 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:49.983607 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:49.983761 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:49.989314 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:49.989338 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:49.989360 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:49.989431 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:49.992894 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.993933 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.993942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.993945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.994106 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:49.994561 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:49.994567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:49.994571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:49.995193 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:49.995558 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:49.995958 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 15:44:49.996301 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 15:44:49.997877 4849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:50.096567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4727
I20260929 15:44:50.117034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4860
Sep 29 15:44:50 dist-test-slave-7491 krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:44:50.138605 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:50.117906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.0.97.3:42811)
0929 15:44:50.118088 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:50.118094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:50.118119 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:44:50.118255 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:50.118261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:50.118343 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:50.118472 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:50.120359 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:50.121830 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:50.123374 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:50.124038 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:50.124231 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:50.129702 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:50.129729 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:50.129753 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:44:50.129816 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:50.134402 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:50.135559 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:50.135571 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:50.135576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:50.135786 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:44:50.136557 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:44:50.136562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:44:50.136565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:44:50.137122 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:50.137502 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:50.137910 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 15:44:50.138162 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 15:44:50.139416 4982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:50.274930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4860
I20260929 15:44:50.296846 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 4502
I20260929 15:44:50.298246 4563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:50.424491 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 4502
2026-09-29T15:44:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5023 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T15:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:50Z Disabled control of system clock
I20260929 15:44:50.506778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42249
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:44:50.901258 5040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:50.901604 5040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:50.901692 5040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:50.910177 5040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:50.910315 5040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:50.910357 5040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:50.910408 5040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:50.910454 5040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:44:50.922039 5040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:50.923460 5040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:50.925490 5040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:50.939965 5046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:50.940307 5045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:50.940359 5048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:50.941831 5040 server_base.cc:1034] running on GCE node
I20260929 15:44:50.943254 5040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:50.946084 5040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:50.947505 5040 hybrid_clock.cc:648] HybridClock initialized: now 1790696690947424 us; error 80 us; skew 500 ppm
I20260929 15:44:50.948138 5040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:50.951846 5040 webserver.cc:460] Webserver started at http://127.0.97.62:33837/ using document root <none> and password file <none>
I20260929 15:44:50.952832 5040 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:50.953008 5040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:50.953389 5040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:50.956832 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "23d5a1d9798045ca9a3057140c85a335"
format_stamp: "Formatted at 2026-09-29 15:44:50 on dist-test-slave-7491"
server_key: "045ba89c0534f7aba0f8ae7d951d4817"
server_key_iv: "c915414e3d083fa328138b8e72a19f2e"
server_key_version: "encryptionkey@0"
I20260929 15:44:50.957801 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "23d5a1d9798045ca9a3057140c85a335"
format_stamp: "Formatted at 2026-09-29 15:44:50 on dist-test-slave-7491"
server_key: "045ba89c0534f7aba0f8ae7d951d4817"
server_key_iv: "c915414e3d083fa328138b8e72a19f2e"
server_key_version: "encryptionkey@0"
I20260929 15:44:50.965348 5040 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 15:44:50.971020 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:50.973022 5040 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 15:44:50.973299 5040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "23d5a1d9798045ca9a3057140c85a335"
format_stamp: "Formatted at 2026-09-29 15:44:50 on dist-test-slave-7491"
server_key: "045ba89c0534f7aba0f8ae7d951d4817"
server_key_iv: "c915414e3d083fa328138b8e72a19f2e"
server_key_version: "encryptionkey@0"
I20260929 15:44:50.973589 5040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:50.980542 5040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:50.981698 5040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:50.982090 5040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:51.005242 5105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42249 every 8 connection(s)
I20260929 15:44:51.005281 5040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42249
I20260929 15:44:51.007303 5040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:51.012817 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:51.013306 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5040
I20260929 15:44:51.013577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:51.014247 388 external_mini_cluster.cc:1428] Setting key 2e7182b62f1edd818ad28457bf37623d
I20260929 15:44:51.025787 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:51.016573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.0.97.62:42249)
0929 15:44:51.017824 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:51.017842 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:51.017900 (+ 58us) server_negotiation.cc:407] Connection header received
0929 15:44:51.020953 (+ 3053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:51.021093 (+ 140us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:51.021835 (+ 742us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:51.022416 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:51.023549 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:51.023558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:51.023576 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:51.023593 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:51.023673 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:51.023700 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:51.023947 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:51.024097 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 15:44:51.024459 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":396,"threads_started":1}
I20260929 15:44:51.029973 5106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335: Bootstrap starting.
I20260929 15:44:51.033711 5106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:51.035147 5106 log.cc:824] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:51.038661 5106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335: No bootstrap required, opened a new log
I20260929 15:44:51.047981 5106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d5a1d9798045ca9a3057140c85a335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42249 } }
I20260929 15:44:51.048511 5106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:51.048625 5106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d5a1d9798045ca9a3057140c85a335, State: Initialized, Role: FOLLOWER
I20260929 15:44:51.049842 5106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d5a1d9798045ca9a3057140c85a335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42249 } }
I20260929 15:44:51.050218 5106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:51.050397 5106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:51.050658 5106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:51.053421 5106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d5a1d9798045ca9a3057140c85a335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42249 } }
I20260929 15:44:51.054149 5106 leader_election.cc:302] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d5a1d9798045ca9a3057140c85a335; no voters:
I20260929 15:44:51.054936 5106 leader_election.cc:288] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:51.055272 5111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:51.056253 5111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [term 1 LEADER]: Becoming Leader. State: Replica: 23d5a1d9798045ca9a3057140c85a335, State: Running, Role: LEADER
I20260929 15:44:51.057101 5111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d5a1d9798045ca9a3057140c85a335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42249 } }
I20260929 15:44:51.058032 5106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:51.060686 5113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d5a1d9798045ca9a3057140c85a335. Latest consensus state: current_term: 1 leader_uuid: "23d5a1d9798045ca9a3057140c85a335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d5a1d9798045ca9a3057140c85a335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42249 } } }
I20260929 15:44:51.060940 5112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23d5a1d9798045ca9a3057140c85a335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d5a1d9798045ca9a3057140c85a335" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42249 } } }
I20260929 15:44:51.061345 5112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:51.061348 5113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:51.062543 5119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:51.069442 5119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:51.081372 5119 catalog_manager.cc:1347] Generated new cluster ID: 33aeebf77fe841d7a94e57ab93848a73
I20260929 15:44:51.081517 5119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:51.121800 5119 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:51.122412 5119 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:51.136013 5119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335: Generated new TSK 0
I20260929 15:44:51.137337 5119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:44:51.147975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42249
--builtin_ntp_servers=127.0.97.20:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:44:51.565383 5130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:51.565754 5130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:51.565923 5130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:51.574683 5130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:51.574844 5130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:51.575044 5130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:44:51.586470 5130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:51.587954 5130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:51.590011 5130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:51.605698 5135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:51.605923 5136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:51.605891 5138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:51.607002 5130 server_base.cc:1034] running on GCE node
I20260929 15:44:51.607820 5130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:51.609025 5130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:51.610278 5130 hybrid_clock.cc:648] HybridClock initialized: now 1790696691610211 us; error 55 us; skew 500 ppm
I20260929 15:44:51.610687 5130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:51.614739 5130 webserver.cc:460] Webserver started at http://127.0.97.1:38165/ using document root <none> and password file <none>
I20260929 15:44:51.615628 5130 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:51.615737 5130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:51.616084 5130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:51.619406 5130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "b37a688aef4947df9d3313dd7e9460b9"
format_stamp: "Formatted at 2026-09-29 15:44:51 on dist-test-slave-7491"
server_key: "316e9cf83de25cce5a8ae2a14d4662e9"
server_key_iv: "4d1e65a012df7a61fe10c2fd986e202c"
server_key_version: "encryptionkey@0"
I20260929 15:44:51.620440 5130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b37a688aef4947df9d3313dd7e9460b9"
format_stamp: "Formatted at 2026-09-29 15:44:51 on dist-test-slave-7491"
server_key: "316e9cf83de25cce5a8ae2a14d4662e9"
server_key_iv: "4d1e65a012df7a61fe10c2fd986e202c"
server_key_version: "encryptionkey@0"
I20260929 15:44:51.626678 5130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:51.631281 5144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:51.633276 5130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:44:51.633524 5130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "b37a688aef4947df9d3313dd7e9460b9"
format_stamp: "Formatted at 2026-09-29 15:44:51 on dist-test-slave-7491"
server_key: "316e9cf83de25cce5a8ae2a14d4662e9"
server_key_iv: "4d1e65a012df7a61fe10c2fd986e202c"
server_key_version: "encryptionkey@0"
I20260929 15:44:51.633809 5130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:51.642095 5130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:51.643076 5130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:51.643378 5130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:51.644532 5130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:51.646360 5130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:51.646495 5130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:51.646627 5130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:51.646714 5130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:51.688074 5130 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43355
I20260929 15:44:51.688125 5256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43355 every 8 connection(s)
I20260929 15:44:51.689595 5130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:44:51.697669 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5130
I20260929 15:44:51.697992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:44:51.698922 388 external_mini_cluster.cc:1428] Setting key 1b44b6d217c876e470a0c88b676c48c3
I20260929 15:44:51.702134 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:51.693434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54035 (local address 127.0.97.62:42249)
0929 15:44:51.694039 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:51.694048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:51.695730 (+ 1682us) server_negotiation.cc:407] Connection header received
0929 15:44:51.697916 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:51.697924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:51.698033 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:51.698375 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:51.701087 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:51.701094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:51.701097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:51.701120 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:51.701183 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:51.701198 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:51.701332 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:51.701736 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 15:44:51.701817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":289,"threads_started":1}
I20260929 15:44:51.702885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42249
--builtin_ntp_servers=127.0.97.20:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:44:51.703293 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:51.694118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42249 (local address 127.0.97.1:54035)
0929 15:44:51.695509 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:51.695561 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:51.697347 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:51.698689 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:51.698700 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:51.698927 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:51.700411 (+ 1484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:51.700414 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:51.700503 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:51.700796 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:51.701323 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:51.701334 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:51.701344 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:44:51.701740 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 15:44:51.702065 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":372,"threads_started":1}
I20260929 15:44:51.705397 5257 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42249
I20260929 15:44:51.705793 5257 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:51.706568 5257 heartbeater.cc:511] Master 127.0.97.62:42249 requested a full tablet report, sending...
I20260929 15:44:51.709708 5071 ts_manager.cc:194] Registered new tserver with Master: b37a688aef4947df9d3313dd7e9460b9 (127.0.97.1:43355)
W20260929 15:44:52.117216 5261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:52.117595 5261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:52.117761 5261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:52.125463 5261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:52.125699 5261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:52.125900 5261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:44:52.135740 5261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:52.137410 5261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:52.139513 5261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:52.155501 5267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:52.155810 5266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:52.155467 5269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:52.156949 5261 server_base.cc:1034] running on GCE node
I20260929 15:44:52.157714 5261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:52.159034 5261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:52.160344 5261 hybrid_clock.cc:648] HybridClock initialized: now 1790696692160267 us; error 57 us; skew 500 ppm
I20260929 15:44:52.160986 5261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:52.164623 5261 webserver.cc:460] Webserver started at http://127.0.97.2:38391/ using document root <none> and password file <none>
I20260929 15:44:52.165717 5261 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:52.165933 5261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:52.166307 5261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:52.169762 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99"
format_stamp: "Formatted at 2026-09-29 15:44:52 on dist-test-slave-7491"
server_key: "6dd31497e237e0c8a797c158994efa8d"
server_key_iv: "89cac985a8dd94a57bd33e7f38811f19"
server_key_version: "encryptionkey@0"
I20260929 15:44:52.170720 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99"
format_stamp: "Formatted at 2026-09-29 15:44:52 on dist-test-slave-7491"
server_key: "6dd31497e237e0c8a797c158994efa8d"
server_key_iv: "89cac985a8dd94a57bd33e7f38811f19"
server_key_version: "encryptionkey@0"
I20260929 15:44:52.177347 5261 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 15:44:52.182405 5275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:52.184453 5261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:44:52.184686 5261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99"
format_stamp: "Formatted at 2026-09-29 15:44:52 on dist-test-slave-7491"
server_key: "6dd31497e237e0c8a797c158994efa8d"
server_key_iv: "89cac985a8dd94a57bd33e7f38811f19"
server_key_version: "encryptionkey@0"
I20260929 15:44:52.185014 5261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:52.191802 5261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:52.192862 5261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:52.193126 5261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:52.194118 5261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:52.195694 5261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:52.195788 5261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:52.195868 5261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:52.195924 5261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:52.234046 5261 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39625
I20260929 15:44:52.234177 5387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39625 every 8 connection(s)
I20260929 15:44:52.235627 5261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:44:52.240255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5261
I20260929 15:44:52.240520 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:44:52.241118 388 external_mini_cluster.cc:1428] Setting key 47f93ebdc81dcae28dbdeb72b364d0a7
I20260929 15:44:52.244710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42249
--builtin_ntp_servers=127.0.97.20:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:44:52.246382 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.239293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42675 (local address 127.0.97.62:42249)
0929 15:44:52.240043 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.240052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.241462 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 15:44:52.243833 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.243839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.243905 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.244015 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.245559 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.245565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.245571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.245591 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.245642 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.245655 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.245797 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.246002 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.246075 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":389,"threads_started":1}
I20260929 15:44:52.247460 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.239966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42249 (local address 127.0.97.2:42675)
0929 15:44:52.241272 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.241327 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.243529 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.244316 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.244326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.244439 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.245217 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.245221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.245264 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.245381 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.245753 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.245765 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.245774 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.246086 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.246398 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":468,"threads_started":1}
I20260929 15:44:52.249006 5388 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42249
I20260929 15:44:52.249343 5388 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:52.250008 5388 heartbeater.cc:511] Master 127.0.97.62:42249 requested a full tablet report, sending...
I20260929 15:44:52.251947 5071 ts_manager.cc:194] Registered new tserver with Master: 6b4edc90ee5e49e1b388e3b85d4ebc99 (127.0.97.2:39625)
W20260929 15:44:52.609184 5392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:52.609573 5392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:52.609728 5392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:52.617688 5392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:52.617859 5392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:52.618036 5392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:44:52.628075 5392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:52.629626 5392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:52.631639 5392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:52.647424 5397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:52.647325 5398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:52.647382 5400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:52.648357 5392 server_base.cc:1034] running on GCE node
I20260929 15:44:52.649084 5392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:52.650238 5392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:52.651511 5392 hybrid_clock.cc:648] HybridClock initialized: now 1790696692651448 us; error 52 us; skew 500 ppm
I20260929 15:44:52.651911 5392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:52.655455 5392 webserver.cc:460] Webserver started at http://127.0.97.3:41907/ using document root <none> and password file <none>
I20260929 15:44:52.656550 5392 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:52.656675 5392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:52.657250 5392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:52.661625 5392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "9d362c57c7fc461b9bd3187c5a8e96c0"
format_stamp: "Formatted at 2026-09-29 15:44:52 on dist-test-slave-7491"
server_key: "aece3d284ee589b8b27055cacd1094cf"
server_key_iv: "b3250dc1e0fb0b07a1c8014388d2f992"
server_key_version: "encryptionkey@0"
I20260929 15:44:52.662839 5392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9d362c57c7fc461b9bd3187c5a8e96c0"
format_stamp: "Formatted at 2026-09-29 15:44:52 on dist-test-slave-7491"
server_key: "aece3d284ee589b8b27055cacd1094cf"
server_key_iv: "b3250dc1e0fb0b07a1c8014388d2f992"
server_key_version: "encryptionkey@0"
I20260929 15:44:52.669346 5392 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:44:52.673619 5406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:52.675416 5392 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 15:44:52.675628 5392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "9d362c57c7fc461b9bd3187c5a8e96c0"
format_stamp: "Formatted at 2026-09-29 15:44:52 on dist-test-slave-7491"
server_key: "aece3d284ee589b8b27055cacd1094cf"
server_key_iv: "b3250dc1e0fb0b07a1c8014388d2f992"
server_key_version: "encryptionkey@0"
I20260929 15:44:52.675892 5392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:52.682375 5392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:52.683333 5392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:52.683631 5392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:52.684839 5392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:44:52.686503 5392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:44:52.686636 5392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:52.686791 5392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:44:52.686877 5392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:52.714068 5257 heartbeater.cc:503] Master 127.0.97.62:42249 was elected leader, sending a full tablet report...
I20260929 15:44:52.728502 5392 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45479
I20260929 15:44:52.728520 5518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45479 every 8 connection(s)
I20260929 15:44:52.730510 5392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:44:52.739162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5392
I20260929 15:44:52.739388 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:44:52.739954 388 external_mini_cluster.cc:1428] Setting key 84e4170264cfa392985a7fe0e73abee5
I20260929 15:44:52.742147 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.734255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41925 (local address 127.0.97.62:42249)
0929 15:44:52.734533 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.734540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.736522 (+ 1982us) server_negotiation.cc:407] Connection header received
0929 15:44:52.738645 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.738650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.738761 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.738929 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.740929 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.740934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.740937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.740957 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.741099 (+ 142us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.741136 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.741396 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.741693 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.741790 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 15:44:52.743697 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.734881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42249 (local address 127.0.97.3:41925)
0929 15:44:52.736273 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.736339 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.738216 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.739345 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.739363 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.739544 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.740510 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.740514 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.740579 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.740714 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.741366 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.741383 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.741395 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.741826 (+ 431us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.742287 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":399,"threads_started":1}
I20260929 15:44:52.745535 5519 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42249
I20260929 15:44:52.745919 5519 heartbeater.cc:464] Registering TS with master...
I20260929 15:44:52.746594 5519 heartbeater.cc:511] Master 127.0.97.62:42249 requested a full tablet report, sending...
I20260929 15:44:52.748545 5071 ts_manager.cc:194] Registered new tserver with Master: 9d362c57c7fc461b9bd3187c5a8e96c0 (127.0.97.3:45479)
I20260929 15:44:52.756932 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:44:52.766054 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.763025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.0.97.62:42249)
0929 15:44:52.763435 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.763444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.763898 (+ 454us) server_negotiation.cc:407] Connection header received
0929 15:44:52.764049 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.764053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.764114 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.764304 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.765456 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.765461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.765464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.765481 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.765519 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.765532 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.765676 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.765713 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.765769 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:44:52.771418 5071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:44:52.779806 5071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:44:52.805132 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.798930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.0.97.1:43355)
0929 15:44:52.800283 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.800292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.800674 (+ 382us) server_negotiation.cc:407] Connection header received
0929 15:44:52.800790 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.800797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.801105 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.801322 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.803951 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.803963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.803981 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.804004 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.804075 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.804098 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.804358 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.804513 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.804648 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":989,"threads_started":1}
I20260929 15:44:52.805226 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.798143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45479 (local address 127.0.0.1:55710)
0929 15:44:52.799358 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.799437 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.799682 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.802910 (+ 3228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.802915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.802942 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.803735 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.803737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.803751 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.803789 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.804416 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.804421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.804426 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.804563 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.804667 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":738,"threads_started":1}
I20260929 15:44:52.806077 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.798232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:51436)
0929 15:44:52.800377 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.800402 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.800523 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.802552 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.802561 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.802680 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.803601 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.803607 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.803650 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.803745 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.804256 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.804269 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.804279 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.805623 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.805671 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1876,"thread_start_us":1928,"threads_started":1}
I20260929 15:44:52.806264 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.798310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.97.3:45479)
0929 15:44:52.800901 (+ 2591us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.800909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.800956 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:44:52.801048 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.801057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.801372 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.801582 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.804012 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.804020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.804030 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.804049 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.804109 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.804130 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.804367 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.805697 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.805805 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2378,"thread_start_us":2167,"threads_started":1}
I20260929 15:44:52.807570 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.800964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.0.97.2:39625)
0929 15:44:52.803154 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.803163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.803197 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:44:52.803254 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.803260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.803536 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.803711 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.806731 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.806739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.806754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.806781 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.806838 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.806855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.807027 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.807109 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.807214 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":1859,"threads_started":1}
I20260929 15:44:52.807623 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.801346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39625 (local address 127.0.0.1:46958)
0929 15:44:52.802145 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.802173 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.802311 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.804696 (+ 2385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.804701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.804731 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.806448 (+ 1717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.806451 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.806470 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.806531 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.806980 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.806985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.806990 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.807128 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.807185 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":411,"threads_started":1}
I20260929 15:44:52.810860 5323 tablet_service.cc:1467] Processing CreateTablet for tablet 99cdb8c35be948849d90d7d7ea376028 (DEFAULT_TABLE table=test-table [id=dcd29521ac0d4c5ea14ea773156549e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:52.810729 5192 tablet_service.cc:1467] Processing CreateTablet for tablet 99cdb8c35be948849d90d7d7ea376028 (DEFAULT_TABLE table=test-table [id=dcd29521ac0d4c5ea14ea773156549e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:52.810724 5454 tablet_service.cc:1467] Processing CreateTablet for tablet 99cdb8c35be948849d90d7d7ea376028 (DEFAULT_TABLE table=test-table [id=dcd29521ac0d4c5ea14ea773156549e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:44:52.812474 5323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cdb8c35be948849d90d7d7ea376028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:52.812790 5454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cdb8c35be948849d90d7d7ea376028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:52.812773 5192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cdb8c35be948849d90d7d7ea376028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:52.826731 5537 tablet_bootstrap.cc:492] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Bootstrap starting.
I20260929 15:44:52.831851 5539 tablet_bootstrap.cc:492] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Bootstrap starting.
I20260929 15:44:52.832433 5537 tablet_bootstrap.cc:654] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:52.832892 5538 tablet_bootstrap.cc:492] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: Bootstrap starting.
I20260929 15:44:52.833925 5537 log.cc:824] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:52.837515 5538 tablet_bootstrap.cc:654] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:52.837709 5539 tablet_bootstrap.cc:654] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:52.838464 5537 tablet_bootstrap.cc:492] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: No bootstrap required, opened a new log
I20260929 15:44:52.838898 5537 ts_tablet_manager.cc:1397] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260929 15:44:52.839459 5538 log.cc:824] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:52.839561 5539 log.cc:824] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:52.844122 5539 tablet_bootstrap.cc:492] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: No bootstrap required, opened a new log
I20260929 15:44:52.844491 5539 ts_tablet_manager.cc:1397] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260929 15:44:52.847090 5538 tablet_bootstrap.cc:492] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: No bootstrap required, opened a new log
I20260929 15:44:52.847527 5538 ts_tablet_manager.cc:1397] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260929 15:44:52.850246 5537 raft_consensus.cc:348] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.850832 5537 raft_consensus.cc:374] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:52.850980 5537 raft_consensus.cc:729] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4edc90ee5e49e1b388e3b85d4ebc99, State: Initialized, Role: FOLLOWER
I20260929 15:44:52.851758 5537 consensus_queue.cc:260] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.853413 5388 heartbeater.cc:503] Master 127.0.97.62:42249 was elected leader, sending a full tablet report...
I20260929 15:44:52.854144 5537 ts_tablet_manager.cc:1428] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 15:44:52.855945 5539 raft_consensus.cc:348] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.856472 5539 raft_consensus.cc:374] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:52.856580 5539 raft_consensus.cc:729] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d362c57c7fc461b9bd3187c5a8e96c0, State: Initialized, Role: FOLLOWER
I20260929 15:44:52.857800 5539 consensus_queue.cc:260] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.858711 5538 raft_consensus.cc:348] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.859297 5538 raft_consensus.cc:374] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:52.859488 5538 raft_consensus.cc:729] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37a688aef4947df9d3313dd7e9460b9, State: Initialized, Role: FOLLOWER
I20260929 15:44:52.860116 5519 heartbeater.cc:503] Master 127.0.97.62:42249 was elected leader, sending a full tablet report...
I20260929 15:44:52.860550 5538 consensus_queue.cc:260] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.862469 5539 ts_tablet_manager.cc:1428] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Time spent starting tablet: real 0.018s user 0.006s sys 0.008s
I20260929 15:44:52.862439 5538 ts_tablet_manager.cc:1428] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260929 15:44:52.926196 5544 raft_consensus.cc:482] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:44:52.926584 5544 raft_consensus.cc:504] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.928542 5544 leader_election.cc:288] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37a688aef4947df9d3313dd7e9460b9 (127.0.97.1:43355), 6b4edc90ee5e49e1b388e3b85d4ebc99 (127.0.97.2:39625)
I20260929 15:44:52.932061 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.929085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.97.3:44855)
0929 15:44:52.929420 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.929449 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.929763 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.930408 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.930412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.930438 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.931218 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.931221 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.931239 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.931291 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.931651 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.931659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.931666 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.931787 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.931855 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 15:44:52.932313 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.929154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44855 (local address 127.0.97.1:43355)
0929 15:44:52.929374 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.929383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.929702 (+ 319us) server_negotiation.cc:407] Connection header received
0929 15:44:52.929974 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.929982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.930084 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.930234 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.931447 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.931456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.931463 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.931487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.931527 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.931549 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.931652 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.931924 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.931995 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:44:52.933046 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.930034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56161 (local address 127.0.97.2:39625)
0929 15:44:52.930276 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.930284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.930497 (+ 213us) server_negotiation.cc:407] Connection header received
0929 15:44:52.930898 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.930904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.930963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.931081 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.932275 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.932281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.932285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.932311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.932354 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.932372 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.932479 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.932692 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.932758 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 15:44:52.933182 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.929568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39625 (local address 127.0.97.3:56161)
0929 15:44:52.930398 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:52.930412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:52.930544 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:52.931113 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:52.931120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:52.931151 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:44:52.931995 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:44:52.931998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:44:52.932015 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:44:52.932062 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:44:52.932574 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:44:52.932579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:44:52.932587 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:44:52.932709 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 15:44:52.932758 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":539,"threads_started":1}
I20260929 15:44:52.933607 5212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cdb8c35be948849d90d7d7ea376028" candidate_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37a688aef4947df9d3313dd7e9460b9" is_pre_election: true
I20260929 15:44:52.934259 5212 raft_consensus.cc:2454] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d362c57c7fc461b9bd3187c5a8e96c0 in term 0.
I20260929 15:44:52.934427 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cdb8c35be948849d90d7d7ea376028" candidate_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" is_pre_election: true
I20260929 15:44:52.934990 5343 raft_consensus.cc:2454] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d362c57c7fc461b9bd3187c5a8e96c0 in term 0.
I20260929 15:44:52.935070 5407 leader_election.cc:302] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d362c57c7fc461b9bd3187c5a8e96c0, b37a688aef4947df9d3313dd7e9460b9; no voters:
I20260929 15:44:52.935514 5544 raft_consensus.cc:2781] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:44:52.935671 5544 raft_consensus.cc:482] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:44:52.935786 5544 raft_consensus.cc:3037] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:52.938171 5544 raft_consensus.cc:504] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.938963 5544 leader_election.cc:288] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [CANDIDATE]: Term 1 election: Requested vote from peers b37a688aef4947df9d3313dd7e9460b9 (127.0.97.1:43355), 6b4edc90ee5e49e1b388e3b85d4ebc99 (127.0.97.2:39625)
I20260929 15:44:52.939553 5212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cdb8c35be948849d90d7d7ea376028" candidate_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37a688aef4947df9d3313dd7e9460b9"
I20260929 15:44:52.939561 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cdb8c35be948849d90d7d7ea376028" candidate_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99"
I20260929 15:44:52.939857 5212 raft_consensus.cc:3037] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:52.939918 5343 raft_consensus.cc:3037] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:52.942354 5343 raft_consensus.cc:2454] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d362c57c7fc461b9bd3187c5a8e96c0 in term 1.
I20260929 15:44:52.942366 5212 raft_consensus.cc:2454] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d362c57c7fc461b9bd3187c5a8e96c0 in term 1.
W20260929 15:44:52.942799 5258 tablet.cc:2369] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:52.942925 5407 leader_election.cc:302] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d362c57c7fc461b9bd3187c5a8e96c0, b37a688aef4947df9d3313dd7e9460b9; no voters:
I20260929 15:44:52.943352 5544 raft_consensus.cc:2781] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:52.943975 5544 raft_consensus.cc:686] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 1 LEADER]: Becoming Leader. State: Replica: 9d362c57c7fc461b9bd3187c5a8e96c0, State: Running, Role: LEADER
I20260929 15:44:52.944586 5544 consensus_queue.cc:237] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } }
I20260929 15:44:52.949769 5071 catalog_manager.cc:5495] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d362c57c7fc461b9bd3187c5a8e96c0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d362c57c7fc461b9bd3187c5a8e96c0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } health_report { overall_health: UNKNOWN } } }
I20260929 15:44:52.976990 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:52.974084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.0.97.3:45479)
0929 15:44:52.974318 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:52.974323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:52.974342 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:44:52.974597 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:52.974602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:52.974690 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:52.974874 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:52.976051 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:52.976056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:52.976060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:52.976078 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:52.976121 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:52.976135 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:52.976365 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:52.976568 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 15:44:52.976634 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 15:44:52.982858 5520 tablet.cc:2369] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:52.987979 5343 raft_consensus.cc:1261] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 1 FOLLOWER]: Refusing update from remote peer 9d362c57c7fc461b9bd3187c5a8e96c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:44:52.988085 5212 raft_consensus.cc:1261] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 1 FOLLOWER]: Refusing update from remote peer 9d362c57c7fc461b9bd3187c5a8e96c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 15:44:52.988075 5389 tablet.cc:2369] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:44:52.989298 5547 consensus_queue.cc:1035] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37a688aef4947df9d3313dd7e9460b9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:52.989789 5544 consensus_queue.cc:1035] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4edc90ee5e49e1b388e3b85d4ebc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:44:53.011845 5551 mvcc.cc:205] Tried to move back new op lower bound from 7334693654466961408 to 7334693654305546240. Current Snapshot: MvccSnapshot[applied={T|T < 7334693654466961408}]
I20260929 15:44:53.037369 5070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:53.037626 5070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:44:53.041783 5070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23d5a1d9798045ca9a3057140c85a335: Sending DeleteTablet for 3 replicas of tablet 99cdb8c35be948849d90d7d7ea376028
I20260929 15:44:53.043322 5192 tablet_service.cc:1514] Processing DeleteTablet for tablet 99cdb8c35be948849d90d7d7ea376028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:53 UTC) from {username='slave'} at 127.0.0.1:51436
I20260929 15:44:53.043743 5323 tablet_service.cc:1514] Processing DeleteTablet for tablet 99cdb8c35be948849d90d7d7ea376028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:53 UTC) from {username='slave'} at 127.0.0.1:46958
I20260929 15:44:53.044124 5454 tablet_service.cc:1514] Processing DeleteTablet for tablet 99cdb8c35be948849d90d7d7ea376028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:44:53 UTC) from {username='slave'} at 127.0.0.1:55710
I20260929 15:44:53.045174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5130
I20260929 15:44:53.061861 5560 tablet_replica.cc:330] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9: stopping tablet replica
I20260929 15:44:53.064366 5560 raft_consensus.cc:2229] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:53.064900 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.048704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.0.97.1:43355)
0929 15:44:53.048970 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:53.048980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:53.058690 (+ 9710us) server_negotiation.cc:407] Connection header received
0929 15:44:53.059497 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:53.059508 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:53.059854 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:53.059996 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:53.063752 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:53.063764 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:53.063774 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:53.063817 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:53.063893 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:53.063921 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:53.064231 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:53.064470 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 15:44:53.064567 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:44:53.065188 5562 tablet_replica.cc:330] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: stopping tablet replica
I20260929 15:44:53.065645 5560 raft_consensus.cc:2258] T 99cdb8c35be948849d90d7d7ea376028 P b37a688aef4947df9d3313dd7e9460b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:53.066140 5252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:53.066833 5562 raft_consensus.cc:2229] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:44:53.070598 5562 raft_consensus.cc:2258] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:53.072419 5561 tablet_replica.cc:330] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: stopping tablet replica
I20260929 15:44:53.072978 5561 raft_consensus.cc:2229] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:44:53.073566 5561 raft_consensus.cc:2258] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:44:53.075839 5561 ts_tablet_manager.cc:1905] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:53.081890 5561 ts_tablet_manager.cc:1918] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:53.082116 5561 log.cc:1196] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/99cdb8c35be948849d90d7d7ea376028
I20260929 15:44:53.082782 5561 ts_tablet_manager.cc:1939] T 99cdb8c35be948849d90d7d7ea376028 P 6b4edc90ee5e49e1b388e3b85d4ebc99: Deleting consensus metadata
I20260929 15:44:53.084367 5562 ts_tablet_manager.cc:1905] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:44:53.085073 5055 catalog_manager.cc:4841] TS 6b4edc90ee5e49e1b388e3b85d4ebc99 (127.0.97.2:39625): tablet 99cdb8c35be948849d90d7d7ea376028 (table test-table [id=dcd29521ac0d4c5ea14ea773156549e9]) successfully deleted
I20260929 15:44:53.090404 5562 ts_tablet_manager.cc:1918] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:44:53.090672 5562 log.cc:1196] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/99cdb8c35be948849d90d7d7ea376028
I20260929 15:44:53.091320 5562 ts_tablet_manager.cc:1939] T 99cdb8c35be948849d90d7d7ea376028 P 9d362c57c7fc461b9bd3187c5a8e96c0: Deleting consensus metadata
I20260929 15:44:53.093091 5057 catalog_manager.cc:4841] TS 9d362c57c7fc461b9bd3187c5a8e96c0 (127.0.97.3:45479): tablet 99cdb8c35be948849d90d7d7ea376028 (table test-table [id=dcd29521ac0d4c5ea14ea773156549e9]) successfully deleted
I20260929 15:44:53.193746 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5130
I20260929 15:44:53.215065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5261
W20260929 15:44:53.215101 5055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43355 (error 108)
I20260929 15:44:53.217412 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.216889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:51454)
0929 15:44:53.217126 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.217266 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 15:44:53.217731 5055 catalog_manager.cc:4584] TS b37a688aef4947df9d3313dd7e9460b9 (127.0.97.1:43355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99cdb8c35be948849d90d7d7ea376028: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
I20260929 15:44:53.219337 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.216439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.97.2:39625)
0929 15:44:53.216619 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:53.216623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:53.216643 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:44:53.216702 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:53.216706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:53.216793 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:53.217038 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:53.218479 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:53.218484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:53.218488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:53.218521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:53.218555 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:53.218568 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:53.218741 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:53.219012 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 15:44:53.219082 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:44:53.220114 5383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:53.271916 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.271206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:51468)
0929 15:44:53.271519 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.271663 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 15:44:53.273558 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.273001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:51474)
0929 15:44:53.273216 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.273340 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:44:53.313612 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5261
I20260929 15:44:53.332285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5392
I20260929 15:44:53.334390 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.333890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:55176)
0929 15:44:53.334088 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.334172 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 15:44:53.336041 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.335646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:55186)
0929 15:44:53.335781 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.335877 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 15:44:53.336125 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.333181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.0.97.3:45479)
0929 15:44:53.333867 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:53.333877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:53.333906 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:44:53.334010 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:53.334016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:53.334094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:53.334233 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:53.335313 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:53.335320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:53.335324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:53.335345 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:53.335394 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:53.335411 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:53.335573 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:53.335702 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 15:44:53.335773 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539}
I20260929 15:44:53.337301 5514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:53.417124 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.416558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:55196)
0929 15:44:53.416823 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.416941 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 15:44:53.418424 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.417999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:55208)
0929 15:44:53.418203 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.418281 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 15:44:53.467547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5392
I20260929 15:44:53.488641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5040
I20260929 15:44:53.490155 5101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:53.601715 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:53.601113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43355 (local address 127.0.0.1:55222)
0929 15:44:53.601385 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:53.601501 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 15:44:53.601719 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5040
2026-09-29T15:44:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3166 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 15:44:53.642556 388 mini_oidc.cc:152] Starting JWKS server
I20260929 15:44:53.645735 388 webserver.cc:460] Webserver started at https://127.0.0.1:32845/ using document root <none> and password file <none>
I20260929 15:44:53.646052 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:44:53.646827 388 webserver.cc:460] Webserver started at http://127.0.0.1:34995/ using document root <none> and password file <none>
2026-09-29T15:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:53Z Disabled control of system clock
I20260929 15:44:53.679752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44897
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44897
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/testchainca.cert with env {}
W20260929 15:44:54.059381 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:54.059752 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:54.059930 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:54.069466 5574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:54.069651 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:54.069710 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:54.069859 5574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:54.069939 5574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:44:54.070034 5574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:44:54.070111 5574 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32845/jwks.json
I20260929 15:44:54.085016 5574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:54.087293 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:54.089375 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:54.103652 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:54.103577 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:54.103629 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:54.105500 5574 server_base.cc:1034] running on GCE node
I20260929 15:44:54.106525 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:54.108438 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:54.109772 5574 hybrid_clock.cc:648] HybridClock initialized: now 1790696694109708 us; error 58 us; skew 500 ppm
I20260929 15:44:54.110486 5574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:54.114022 5574 webserver.cc:460] Webserver started at http://127.0.97.62:41401/ using document root <none> and password file <none>
I20260929 15:44:54.114892 5574 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:54.115046 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:54.115377 5574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:54.118722 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "c04114423f4f411e8e2d8db66cf89e5c"
format_stamp: "Formatted at 2026-09-29 15:44:54 on dist-test-slave-7491"
server_key: "754635a7815431870e1fc04af58ec7db"
server_key_iv: "367d9ca0b7a493de54b4dff371054427"
server_key_version: "encryptionkey@0"
I20260929 15:44:54.119649 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "c04114423f4f411e8e2d8db66cf89e5c"
format_stamp: "Formatted at 2026-09-29 15:44:54 on dist-test-slave-7491"
server_key: "754635a7815431870e1fc04af58ec7db"
server_key_iv: "367d9ca0b7a493de54b4dff371054427"
server_key_version: "encryptionkey@0"
I20260929 15:44:54.125887 5574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 15:44:54.130029 5588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:54.131692 5574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:44:54.131963 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "c04114423f4f411e8e2d8db66cf89e5c"
format_stamp: "Formatted at 2026-09-29 15:44:54 on dist-test-slave-7491"
server_key: "754635a7815431870e1fc04af58ec7db"
server_key_iv: "367d9ca0b7a493de54b4dff371054427"
server_key_version: "encryptionkey@0"
I20260929 15:44:54.132287 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:54.158885 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:54.159916 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:54.160305 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:54.182842 5574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44897
I20260929 15:44:54.182888 5639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44897 every 8 connection(s)
I20260929 15:44:54.184876 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:54.189774 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:54.195055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5574
I20260929 15:44:54.195269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:54.195912 388 external_mini_cluster.cc:1428] Setting key 5f6c1f8dab7e1bad2435ea60dfa4edf1
I20260929 15:44:54.202111 5640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c: Bootstrap starting.
I20260929 15:44:54.206274 5640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:54.207669 5640 log.cc:824] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:54.211462 5640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c: No bootstrap required, opened a new log
I20260929 15:44:54.214085 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:54.198375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.0.97.62:44897)
0929 15:44:54.199676 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:54.199692 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:54.199744 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:44:54.201760 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:54.201786 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:54.202784 (+ 998us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:54.203299 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:54.204162 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:54.205740 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:54.207764 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:54.208978 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:54.209214 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:54.210816 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:54.210827 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:54.210851 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:54.211894 (+ 1043us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:54.211996 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:54.212028 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:54.212144 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:54.212607 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 15:44:54.212976 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":503,"threads_started":1}
I20260929 15:44:54.219544 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:44:54.222375 5640 raft_consensus.cc:348] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04114423f4f411e8e2d8db66cf89e5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260929 15:44:54.222981 5640 raft_consensus.cc:374] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:54.223115 5640 raft_consensus.cc:729] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04114423f4f411e8e2d8db66cf89e5c, State: Initialized, Role: FOLLOWER
I20260929 15:44:54.223989 5640 consensus_queue.cc:260] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04114423f4f411e8e2d8db66cf89e5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260929 15:44:54.224357 5640 raft_consensus.cc:388] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:54.224519 5640 raft_consensus.cc:482] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:54.224889 5640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:54.227319 5640 raft_consensus.cc:504] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04114423f4f411e8e2d8db66cf89e5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260929 15:44:54.227963 5640 leader_election.cc:302] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c04114423f4f411e8e2d8db66cf89e5c; no voters:
I20260929 15:44:54.228765 5640 leader_election.cc:288] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:54.228888 5645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:54.229604 5645 raft_consensus.cc:686] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [term 1 LEADER]: Becoming Leader. State: Replica: c04114423f4f411e8e2d8db66cf89e5c, State: Running, Role: LEADER
I20260929 15:44:54.230261 5645 consensus_queue.cc:237] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04114423f4f411e8e2d8db66cf89e5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260929 15:44:54.231740 5640 sys_catalog.cc:564] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:54.234555 5646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c04114423f4f411e8e2d8db66cf89e5c. Latest consensus state: current_term: 1 leader_uuid: "c04114423f4f411e8e2d8db66cf89e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04114423f4f411e8e2d8db66cf89e5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } } }
I20260929 15:44:54.234850 5647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c04114423f4f411e8e2d8db66cf89e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c04114423f4f411e8e2d8db66cf89e5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } } }
I20260929 15:44:54.235224 5646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:54.235289 5647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:54.237511 5652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:54.243441 5652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:54.256626 5652 catalog_manager.cc:1347] Generated new cluster ID: 62d95fce13f54f6eb51cb3e6682be1be
I20260929 15:44:54.256814 5652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:54.286710 5652 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:54.288656 5652 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:54.310771 5652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c04114423f4f411e8e2d8db66cf89e5c: Generated new TSK 0
I20260929 15:44:54.312371 5652 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:44:54.371230 5670 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 15:44:54.371445 5670 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:44:54.372570 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:54.346108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.0.97.62:44897)
0929 15:44:54.346392 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:54.346401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:54.347376 (+ 975us) server_negotiation.cc:407] Connection header received
0929 15:44:54.347988 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:54.347992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:54.348085 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:54.348473 (+ 388us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:44:54.349338 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:54.350184 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:54.352683 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:54.353477 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:54.353656 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:54.354652 (+ 996us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:44:54.371893 (+ 17241us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:44:54.372331 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 15:44:54.372735 5643 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 15:44:54.375466 5669 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:44:54.345801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.0.1:50078)
0929 15:44:54.347131 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:54.347168 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:54.347499 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:54.348522 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:54.348528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:54.348549 (+ 21us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:44:54.349058 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:54.349072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:54.350469 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:54.350478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:54.352405 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:54.352442 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:54.353713 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:54.353794 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:54.354033 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:54.354146 (+ 113us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:44:54.372394 (+ 18248us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:44:54.372985 (+ 591us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":783,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:44897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 15:44:54.377789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5574
I20260929 15:44:54.379379 5635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:54.488534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5574
2026-09-29T15:44:54Z chronyd exiting
I20260929 15:44:54.543672 388 test_util.cc:175] -----------------------------------------------
I20260929 15:44:54.543926 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790696645767654-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (906 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 15:44:54.548034 388 mini_oidc.cc:152] Starting JWKS server
I20260929 15:44:54.550074 388 webserver.cc:460] Webserver started at https://127.0.0.1:44787/ using document root <none> and password file <none>
I20260929 15:44:54.550383 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:44:54.551187 388 webserver.cc:460] Webserver started at http://127.0.0.1:38489/ using document root <none> and password file <none>
2026-09-29T15:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:54Z Disabled control of system clock
I20260929 15:44:54.583946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35395
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35395
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44787/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/testchainca.cert with env {}
W20260929 15:44:54.976130 5677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:54.976608 5677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:54.976784 5677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:54.985225 5677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:54.985469 5677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:54.985553 5677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:54.985666 5677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:54.985751 5677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:44:54.985862 5677 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:44:54.985936 5677 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44787/jwks.json
I20260929 15:44:54.997926 5677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44787/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:54.999529 5677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:55.001065 5677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:55.016388 5685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:55.016507 5682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:55.016497 5683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:55.017732 5677 server_base.cc:1034] running on GCE node
I20260929 15:44:55.018901 5677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:55.021183 5677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:55.022511 5677 hybrid_clock.cc:648] HybridClock initialized: now 1790696695022459 us; error 51 us; skew 500 ppm
I20260929 15:44:55.023025 5677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:55.033627 5677 webserver.cc:460] Webserver started at http://127.0.97.62:43051/ using document root <none> and password file <none>
I20260929 15:44:55.034721 5677 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:55.034847 5677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:55.035246 5677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:55.039306 5677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34"
format_stamp: "Formatted at 2026-09-29 15:44:55 on dist-test-slave-7491"
server_key: "ab9ab71e6c4ae38b7951086c55cc2d7b"
server_key_iv: "a2042ed19d357b21f6ed33e53ef252ce"
server_key_version: "encryptionkey@0"
I20260929 15:44:55.040509 5677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34"
format_stamp: "Formatted at 2026-09-29 15:44:55 on dist-test-slave-7491"
server_key: "ab9ab71e6c4ae38b7951086c55cc2d7b"
server_key_iv: "a2042ed19d357b21f6ed33e53ef252ce"
server_key_version: "encryptionkey@0"
I20260929 15:44:55.046953 5677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:44:55.051311 5691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:55.053220 5677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:44:55.053465 5677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34"
format_stamp: "Formatted at 2026-09-29 15:44:55 on dist-test-slave-7491"
server_key: "ab9ab71e6c4ae38b7951086c55cc2d7b"
server_key_iv: "a2042ed19d357b21f6ed33e53ef252ce"
server_key_version: "encryptionkey@0"
I20260929 15:44:55.053689 5677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:55.078863 5677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:55.080199 5677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:55.080560 5677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:55.104627 5742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35395 every 8 connection(s)
I20260929 15:44:55.104612 5677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35395
I20260929 15:44:55.106606 5677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:55.110543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5677
I20260929 15:44:55.110803 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:55.111433 388 external_mini_cluster.cc:1428] Setting key 81b09d344660c9a1537b22467fe60751
I20260929 15:44:55.111431 5743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:55.127039 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34: Bootstrap starting.
I20260929 15:44:55.128083 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:55.113459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.0.97.62:35395)
0929 15:44:55.114737 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:55.114749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:55.114801 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:44:55.116694 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:55.116722 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:55.117305 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:55.117732 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:55.118473 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:55.119925 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:55.121981 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:55.123030 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:55.123274 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:55.125038 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:55.125049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:55.125069 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:55.125803 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:55.125906 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:55.125935 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:55.126061 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:55.126649 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 15:44:55.126983 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":503,"threads_started":1}
I20260929 15:44:55.130626 5743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:55.132081 5743 log.cc:824] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:55.132705 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:44:55.136137 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34: No bootstrap required, opened a new log
I20260929 15:44:55.145843 5743 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35395 } }
I20260929 15:44:55.146428 5743 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:55.146566 5743 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfff8085d4f6461c8e83c5ea0b9e8b34, State: Initialized, Role: FOLLOWER
I20260929 15:44:55.147297 5743 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35395 } }
I20260929 15:44:55.147569 5743 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:55.147773 5743 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:55.147982 5743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:55.151156 5743 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35395 } }
I20260929 15:44:55.152141 5743 leader_election.cc:302] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfff8085d4f6461c8e83c5ea0b9e8b34; no voters:
I20260929 15:44:55.152992 5743 leader_election.cc:288] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:55.153164 5748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:55.154340 5748 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [term 1 LEADER]: Becoming Leader. State: Replica: cfff8085d4f6461c8e83c5ea0b9e8b34, State: Running, Role: LEADER
I20260929 15:44:55.154937 5748 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35395 } }
I20260929 15:44:55.155496 5743 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:55.158597 5749 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35395 } } }
I20260929 15:44:55.158635 5750 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfff8085d4f6461c8e83c5ea0b9e8b34. Latest consensus state: current_term: 1 leader_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfff8085d4f6461c8e83c5ea0b9e8b34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35395 } } }
I20260929 15:44:55.159085 5749 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:55.159116 5750 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:55.161001 5754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:55.167408 5754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:55.179360 5754 catalog_manager.cc:1347] Generated new cluster ID: 3152131358a04550a54edd195d5d6944
I20260929 15:44:55.179589 5754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:55.193564 5754 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:55.196839 5754 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:55.210429 5754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfff8085d4f6461c8e83c5ea0b9e8b34: Generated new TSK 0
I20260929 15:44:55.211503 5754 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:44:55.254149 5773 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 15:44:55.254293 5773 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:44:55.255795 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:55.233435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.0.97.62:35395)
0929 15:44:55.233702 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:55.233713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:55.234350 (+ 637us) server_negotiation.cc:407] Connection header received
0929 15:44:55.234646 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:55.234652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:55.234751 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:55.234921 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:44:55.235540 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:55.236440 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:55.238844 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:55.239310 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:55.239429 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:55.240241 (+ 812us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:44:55.255059 (+ 14818us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:44:55.255511 (+ 452us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57468: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 15:44:55.255957 5746 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57468: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 15:44:55.256129 5772 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:44:55.233405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35395 (local address 127.0.0.1:57468)
0929 15:44:55.234149 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:55.234171 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:55.234371 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:55.234963 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:55.234968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:55.234982 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:44:55.235335 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:55.235349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:55.236599 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:55.236605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:55.238602 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:55.238621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:55.239454 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:55.239508 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:55.239733 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:55.239786 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:44:55.255362 (+ 15576us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:44:55.255667 (+ 305us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":428,"threads_started":1}
I20260929 15:44:55.257962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5677
I20260929 15:44:55.259719 5738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:55.365447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5677
2026-09-29T15:44:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (880 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 15:44:55.428529 388 mini_oidc.cc:152] Starting JWKS server
I20260929 15:44:55.430652 388 webserver.cc:460] Webserver started at https://127.0.0.1:46127/ using document root <none> and password file <none>
I20260929 15:44:55.430977 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:44:55.431866 388 webserver.cc:460] Webserver started at http://127.0.0.1:32935/ using document root <none> and password file <none>
2026-09-29T15:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:55Z Disabled control of system clock
I20260929 15:44:55.465903 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44597
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46127/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:44:55.868551 5780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:55.868901 5780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:55.869072 5780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:55.879895 5780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:55.880070 5780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:55.880115 5780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:55.880249 5780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:55.880317 5780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:44:55.880383 5780 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:44:55.880426 5780 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46127/jwks.json
I20260929 15:44:55.891930 5780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46127/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:55.893487 5780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:55.895927 5780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:55.912950 5785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:55.913111 5786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:55.913115 5788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:55.914531 5780 server_base.cc:1034] running on GCE node
I20260929 15:44:55.915539 5780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:55.917477 5780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:55.918790 5780 hybrid_clock.cc:648] HybridClock initialized: now 1790696695918732 us; error 55 us; skew 500 ppm
I20260929 15:44:55.919234 5780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:55.922765 5780 webserver.cc:460] Webserver started at http://127.0.97.62:35861/ using document root <none> and password file <none>
I20260929 15:44:55.923780 5780 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:55.923942 5780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:55.924432 5780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:55.928238 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "149256878de648f9ae19852daa5c1136"
format_stamp: "Formatted at 2026-09-29 15:44:55 on dist-test-slave-7491"
server_key: "d9b57ebfc3a26fff9fdd2bf7a7c49916"
server_key_iv: "23adf0541687fc7418c0b1990f5291ce"
server_key_version: "encryptionkey@0"
I20260929 15:44:55.929248 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "149256878de648f9ae19852daa5c1136"
format_stamp: "Formatted at 2026-09-29 15:44:55 on dist-test-slave-7491"
server_key: "d9b57ebfc3a26fff9fdd2bf7a7c49916"
server_key_iv: "23adf0541687fc7418c0b1990f5291ce"
server_key_version: "encryptionkey@0"
I20260929 15:44:55.936951 5780 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260929 15:44:55.941555 5794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:55.943419 5780 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:44:55.943686 5780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "149256878de648f9ae19852daa5c1136"
format_stamp: "Formatted at 2026-09-29 15:44:55 on dist-test-slave-7491"
server_key: "d9b57ebfc3a26fff9fdd2bf7a7c49916"
server_key_iv: "23adf0541687fc7418c0b1990f5291ce"
server_key_version: "encryptionkey@0"
I20260929 15:44:55.943974 5780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:55.961781 5780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:55.962785 5780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:55.963112 5780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:55.986626 5780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44597
I20260929 15:44:55.986656 5845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44597 every 8 connection(s)
I20260929 15:44:55.988931 5780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:55.994232 5846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:55.997810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5780
I20260929 15:44:55.998024 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:55.998992 388 external_mini_cluster.cc:1428] Setting key f39f5495e98845d5b5f701dd8deeb33c
I20260929 15:44:56.009423 5846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136: Bootstrap starting.
I20260929 15:44:56.013250 5846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:56.015110 5846 log.cc:824] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:56.016292 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:56.001694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.97.62:44597)
0929 15:44:56.003140 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:56.003156 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:56.003224 (+ 68us) server_negotiation.cc:407] Connection header received
0929 15:44:56.005342 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:56.005374 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:56.006085 (+ 711us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:56.006688 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:56.007688 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.009269 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.010959 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.011997 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.012273 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:56.013523 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:56.013531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:56.013550 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:56.014212 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:56.014327 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:56.014352 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:56.014504 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:56.014748 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 15:44:56.015119 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":504,"threads_started":1}
I20260929 15:44:56.018846 5846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136: No bootstrap required, opened a new log
I20260929 15:44:56.021620 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:44:56.029876 5846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149256878de648f9ae19852daa5c1136" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44597 } }
I20260929 15:44:56.030306 5846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:56.030380 5846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149256878de648f9ae19852daa5c1136, State: Initialized, Role: FOLLOWER
I20260929 15:44:56.031100 5846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149256878de648f9ae19852daa5c1136" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44597 } }
I20260929 15:44:56.031430 5846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:56.031608 5846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:56.031814 5846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:56.034312 5846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149256878de648f9ae19852daa5c1136" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44597 } }
I20260929 15:44:56.034915 5846 leader_election.cc:302] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 149256878de648f9ae19852daa5c1136; no voters:
I20260929 15:44:56.035583 5846 leader_election.cc:288] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:56.035840 5851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:56.036875 5851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [term 1 LEADER]: Becoming Leader. State: Replica: 149256878de648f9ae19852daa5c1136, State: Running, Role: LEADER
I20260929 15:44:56.037456 5851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149256878de648f9ae19852daa5c1136" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44597 } }
I20260929 15:44:56.038429 5846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:56.041098 5853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 149256878de648f9ae19852daa5c1136. Latest consensus state: current_term: 1 leader_uuid: "149256878de648f9ae19852daa5c1136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149256878de648f9ae19852daa5c1136" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44597 } } }
I20260929 15:44:56.041500 5853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:56.041038 5852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "149256878de648f9ae19852daa5c1136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149256878de648f9ae19852daa5c1136" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44597 } } }
I20260929 15:44:56.042245 5852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:56.046113 5858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:56.054106 5858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:56.065864 5858 catalog_manager.cc:1347] Generated new cluster ID: aedc3da9814d472eb0ef6c9450c79bfc
I20260929 15:44:56.066013 5858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:56.083012 5858 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:56.084815 5858 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:56.098415 5858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 149256878de648f9ae19852daa5c1136: Generated new TSK 0
I20260929 15:44:56.100059 5858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:44:56.131815 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:56.120415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.0.97.62:44597)
0929 15:44:56.120703 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:56.120713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:56.121248 (+ 535us) server_negotiation.cc:407] Connection header received
0929 15:44:56.121595 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:56.121600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:56.121698 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:56.121914 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:44:56.122704 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.123564 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.126319 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.126875 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.127023 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:56.128265 (+ 1242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:44:56.131075 (+ 2810us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:44:56.131564 (+ 489us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":124}
W20260929 15:44:56.131982 5849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 15:44:56.135351 5875 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:44:56.120032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44597 (local address 127.0.0.1:51790)
0929 15:44:56.121012 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:56.121041 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:56.121280 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:56.121969 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:56.121976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:56.121993 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:44:56.122401 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:56.122415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.123823 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.123832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:56.126018 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:56.126049 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.127138 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.127257 (+ 119us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:56.127541 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:56.127725 (+ 184us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:44:56.131388 (+ 3663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:44:56.134890 (+ 3502us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46127/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":3047,"thread_start_us":478,"threads_started":1}
I20260929 15:44:56.138212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5780
I20260929 15:44:56.139909 5841 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 15:44:56.145663 5876 webserver.cc:533] Webserver: sslize failed:
E20260929 15:44:56.145783 5876 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:44:56.250339 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5780
2026-09-29T15:44:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (881 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T15:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:56Z Disabled control of system clock
I20260929 15:44:56.342633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38011
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38011
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/testchainca.cert with env {}
W20260929 15:44:56.759600 5881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:56.759972 5881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:56.760072 5881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:44:56.768594 5881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:44:56.768748 5881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:44:56.768788 5881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:44:56.768847 5881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:44:56.768886 5881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:44:56.768952 5881 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:44:56.769011 5881 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 15:44:56.781155 5881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:44:56.783213 5881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:44:56.784897 5881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:44:56.799757 5886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:56.799774 5887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:44:56.799796 5889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:44:56.801801 5881 server_base.cc:1034] running on GCE node
I20260929 15:44:56.802812 5881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:44:56.804677 5881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:44:56.805971 5881 hybrid_clock.cc:648] HybridClock initialized: now 1790696696805928 us; error 68 us; skew 500 ppm
I20260929 15:44:56.806417 5881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:44:56.810002 5881 webserver.cc:460] Webserver started at http://127.0.97.62:44693/ using document root <none> and password file <none>
I20260929 15:44:56.810889 5881 fs_manager.cc:359] Metadata directory not provided
I20260929 15:44:56.811060 5881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:44:56.811416 5881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:44:56.814692 5881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "c5cacb5648ba4b9ebbb573a0ec521b27"
format_stamp: "Formatted at 2026-09-29 15:44:56 on dist-test-slave-7491"
server_key: "408a27a3b57eac9c26af6b3b17f5f9f1"
server_key_iv: "cc0a8954135f78f520e23a329a785dfc"
server_key_version: "encryptionkey@0"
I20260929 15:44:56.815634 5881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5cacb5648ba4b9ebbb573a0ec521b27"
format_stamp: "Formatted at 2026-09-29 15:44:56 on dist-test-slave-7491"
server_key: "408a27a3b57eac9c26af6b3b17f5f9f1"
server_key_iv: "cc0a8954135f78f520e23a329a785dfc"
server_key_version: "encryptionkey@0"
I20260929 15:44:56.822757 5881 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:44:56.827034 5895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:44:56.828635 5881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:44:56.828856 5881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "c5cacb5648ba4b9ebbb573a0ec521b27"
format_stamp: "Formatted at 2026-09-29 15:44:56 on dist-test-slave-7491"
server_key: "408a27a3b57eac9c26af6b3b17f5f9f1"
server_key_iv: "cc0a8954135f78f520e23a329a785dfc"
server_key_version: "encryptionkey@0"
I20260929 15:44:56.829098 5881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:44:56.845381 5881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:44:56.846391 5881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:44:56.846705 5881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:44:56.869510 5881 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38011
I20260929 15:44:56.869511 5946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38011 every 8 connection(s)
I20260929 15:44:56.871147 5881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:44:56.873281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 5881
I20260929 15:44:56.873518 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:44:56.874358 388 external_mini_cluster.cc:1428] Setting key 6aa00d899f5486b60c8541113ddfd3db
I20260929 15:44:56.876881 5947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:44:56.891371 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:56.877397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.97.62:38011)
0929 15:44:56.878710 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:56.878724 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:56.878784 (+ 60us) server_negotiation.cc:407] Connection header received
0929 15:44:56.880826 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:56.880860 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:56.881474 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:56.881915 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:44:56.882741 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.884293 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.886245 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:56.887054 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:56.887210 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:56.888710 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:44:56.888718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:44:56.888736 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:44:56.889387 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:44:56.889495 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:44:56.889523 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:44:56.889629 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 15:44:56.889883 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 15:44:56.890303 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":460,"threads_started":1}
I20260929 15:44:56.892373 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27: Bootstrap starting.
I20260929 15:44:56.897156 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:44:56.897301 5947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27: Neither blocks nor log segments found. Creating new log.
I20260929 15:44:56.898754 5947 log.cc:824] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27: Log is configured to *not* fsync() on all Append() calls
I20260929 15:44:56.903549 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27: No bootstrap required, opened a new log
I20260929 15:44:56.912039 5947 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38011 } }
I20260929 15:44:56.912607 5947 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:44:56.912851 5947 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cacb5648ba4b9ebbb573a0ec521b27, State: Initialized, Role: FOLLOWER
I20260929 15:44:56.913857 5947 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38011 } }
I20260929 15:44:56.914188 5947 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:44:56.914356 5947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:44:56.914563 5947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:44:56.917177 5947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38011 } }
I20260929 15:44:56.917838 5947 leader_election.cc:302] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5cacb5648ba4b9ebbb573a0ec521b27; no voters:
I20260929 15:44:56.918573 5947 leader_election.cc:288] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:44:56.918898 5952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:44:56.919814 5952 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [term 1 LEADER]: Becoming Leader. State: Replica: c5cacb5648ba4b9ebbb573a0ec521b27, State: Running, Role: LEADER
I20260929 15:44:56.920825 5952 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38011 } }
I20260929 15:44:56.922289 5947 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:44:56.925640 5954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5cacb5648ba4b9ebbb573a0ec521b27. Latest consensus state: current_term: 1 leader_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38011 } } }
I20260929 15:44:56.926061 5954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:56.926450 5953 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cacb5648ba4b9ebbb573a0ec521b27" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38011 } } }
I20260929 15:44:56.926818 5953 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27 [sys.catalog]: This master's current role is: LEADER
I20260929 15:44:56.929226 5958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:44:56.935254 5958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:44:56.949195 5958 catalog_manager.cc:1347] Generated new cluster ID: 8e25d67aeeee44d99babbb9546a8281a
I20260929 15:44:56.949501 5958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:44:56.961277 5958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:44:56.963376 5958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:44:56.975189 5958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5cacb5648ba4b9ebbb573a0ec521b27: Generated new TSK 0
I20260929 15:44:56.976508 5958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:44:57.031975 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:44:56.999437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.0.97.62:38011)
0929 15:44:56.999749 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 15:44:56.999761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:44:57.000426 (+ 665us) server_negotiation.cc:407] Connection header received
0929 15:44:57.000751 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:44:57.000939 (+ 188us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:44:57.001101 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:44:57.001324 (+ 223us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:44:57.001971 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:57.002862 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:57.005078 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:44:57.005606 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:44:57.005761 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:57.006463 (+ 702us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:44:57.031163 (+ 24700us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:44:57.031642 (+ 479us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51330: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 15:44:57.032126 5950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51330: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 15:44:57.032287 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:44:56.999384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38011 (local address 127.0.0.1:51330)
0929 15:44:57.000273 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 15:44:57.000297 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:44:57.000482 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:44:57.001366 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:44:57.001372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:44:57.001388 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:44:57.001725 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:57.001739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:57.003062 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:57.003069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:57.004836 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:44:57.004857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:44:57.005776 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:44:57.005856 (+ 80us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:44:57.006074 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:44:57.006130 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:44:57.031567 (+ 25437us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:44:57.031828 (+ 261us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38011: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":468,"threads_started":1}
I20260929 15:44:57.034173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 5881
I20260929 15:44:57.035495 5942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:44:57.141765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 5881
2026-09-29T15:44:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (873 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (498 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (506 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:44:58 dist-test-slave-7491 krb5kdc[5991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:44:58 dist-test-slave-7491 krb5kdc[5991](info): set up 2 sockets
Sep 29 15:44:58 dist-test-slave-7491 krb5kdc[5991](info): commencing operation
krb5kdc: starting...
W20260929 15:44:59.317912 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:44:59 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:44:59.577611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46101
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:44:59.991890 6004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:44:59.992368 6004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:44:59.992556 6004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:00.002635 6004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:00.002921 6004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:00.003026 6004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:00.003116 6004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:00.003185 6004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:00.015856 6004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:00.017980 6004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:00.019698 6004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:00.035006 6009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:00.035056 6010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:00.035125 6012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:00.041918 6004 server_base.cc:1034] running on GCE node
I20260929 15:45:00.043054 6004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:00.044966 6004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:00.046258 6004 hybrid_clock.cc:648] HybridClock initialized: now 1790696700046205 us; error 48 us; skew 500 ppm
I20260929 15:45:00.046720 6004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:00 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:00.057451 6004 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:45:00.059556 6004 webserver.cc:460] Webserver started at http://127.0.97.62:39661/ using document root <none> and password file <none>
I20260929 15:45:00.060534 6004 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:00.060698 6004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:00.061048 6004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:00.064633 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "93efa8d5e81c4e799e8fdb40cf6e78be"
format_stamp: "Formatted at 2026-09-29 15:45:00 on dist-test-slave-7491"
server_key: "2914386e0c0e49ff06ee7e805bc31898"
server_key_iv: "9158d3f8464526d0dc2511d61f9d93de"
server_key_version: "encryptionkey@0"
I20260929 15:45:00.065548 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "93efa8d5e81c4e799e8fdb40cf6e78be"
format_stamp: "Formatted at 2026-09-29 15:45:00 on dist-test-slave-7491"
server_key: "2914386e0c0e49ff06ee7e805bc31898"
server_key_iv: "9158d3f8464526d0dc2511d61f9d93de"
server_key_version: "encryptionkey@0"
I20260929 15:45:00.072007 6004 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 15:45:00.077281 6019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:00.079448 6004 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 15:45:00.079749 6004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "93efa8d5e81c4e799e8fdb40cf6e78be"
format_stamp: "Formatted at 2026-09-29 15:45:00 on dist-test-slave-7491"
server_key: "2914386e0c0e49ff06ee7e805bc31898"
server_key_iv: "9158d3f8464526d0dc2511d61f9d93de"
server_key_version: "encryptionkey@0"
I20260929 15:45:00.080220 6004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:00.110679 6004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:00.117223 6004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:00.117651 6004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:00.142251 6004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46101
I20260929 15:45:00.142271 6070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46101 every 8 connection(s)
I20260929 15:45:00.143884 6004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:00.147221 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6004
I20260929 15:45:00.147540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:00.148342 388 external_mini_cluster.cc:1428] Setting key 033e1244262463d52cc454aa71e932b2
I20260929 15:45:00.148732 6071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:00.165984 6071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be: Bootstrap starting.
Sep 29 15:45:00 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:00.170369 6071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:00.171772 6071 log.cc:824] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:00.175197 6071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be: No bootstrap required, opened a new log
I20260929 15:45:00.179131 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:00.151414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.0.97.62:46101)
0929 15:45:00.152987 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:00.153009 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:00.153080 (+ 71us) server_negotiation.cc:407] Connection header received
0929 15:45:00.155056 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:00.155095 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:00.155794 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:00.156428 (+ 634us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:00.158412 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:00.160597 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:00.162324 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:00.163605 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:00.163877 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:00.169414 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:00.169436 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:00.169487 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:00.169581 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:00.173654 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:00.174580 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:00.174588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:00.174595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:00.174785 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:00.175224 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:00.175229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:00.175237 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:00.175877 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:00.176257 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:00.176837 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 15:45:00.177364 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":586,"threads_started":1}
I20260929 15:45:00.185933 6071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46101 } }
I20260929 15:45:00.186386 6071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:00.186645 6071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93efa8d5e81c4e799e8fdb40cf6e78be, State: Initialized, Role: FOLLOWER
I20260929 15:45:00.187480 6071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46101 } }
I20260929 15:45:00.187772 6071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:00.187934 6071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:00.188133 6071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:00.190768 6071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46101 } }
I20260929 15:45:00.191442 6071 leader_election.cc:302] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93efa8d5e81c4e799e8fdb40cf6e78be; no voters:
I20260929 15:45:00.192240 6071 leader_election.cc:288] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:00.192420 6076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:00.193856 6076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [term 1 LEADER]: Becoming Leader. State: Replica: 93efa8d5e81c4e799e8fdb40cf6e78be, State: Running, Role: LEADER
I20260929 15:45:00.194880 6076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46101 } }
I20260929 15:45:00.195675 6071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:00.199589 6078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93efa8d5e81c4e799e8fdb40cf6e78be. Latest consensus state: current_term: 1 leader_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46101 } } }
I20260929 15:45:00.200448 6078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:00.201700 6083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:00.201577 6077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93efa8d5e81c4e799e8fdb40cf6e78be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46101 } } }
I20260929 15:45:00.202448 6077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:00.208351 6083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:00.222612 6083 catalog_manager.cc:1347] Generated new cluster ID: dedd5185f31d48efb6dbde4eeff23058
I20260929 15:45:00.222820 6083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:00.251116 6083 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:00.252652 6083 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:00.265949 6083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93efa8d5e81c4e799e8fdb40cf6e78be: Generated new TSK 0
I20260929 15:45:00.267036 6083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:45:00.350816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46101
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:00.791853 6097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:00.792249 6097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:00.792411 6097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:00.801822 6097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:00.801993 6097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:00.802294 6097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:00.816222 6097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:00.817770 6097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:00.820133 6097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:00.837724 6102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:00.838726 6105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:00.838764 6103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:00.839872 6097 server_base.cc:1034] running on GCE node
I20260929 15:45:00.841246 6097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:00.842950 6097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:00.844370 6097 hybrid_clock.cc:648] HybridClock initialized: now 1790696700844271 us; error 79 us; skew 500 ppm
I20260929 15:45:00.844798 6097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:00 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:00.848449 6097 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:45:00.850215 6097 webserver.cc:460] Webserver started at http://127.0.97.1:41871/ using document root <none> and password file <none>
I20260929 15:45:00.851038 6097 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:00.851159 6097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:00.851440 6097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:00.854875 6097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "5a3904efb3514162a0447220f4110dc4"
format_stamp: "Formatted at 2026-09-29 15:45:00 on dist-test-slave-7491"
server_key: "8753ac073754a6d30b4374b8e273e378"
server_key_iv: "8ca3c2b62189d76f5ea104d1bc13ab18"
server_key_version: "encryptionkey@0"
I20260929 15:45:00.855738 6097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5a3904efb3514162a0447220f4110dc4"
format_stamp: "Formatted at 2026-09-29 15:45:00 on dist-test-slave-7491"
server_key: "8753ac073754a6d30b4374b8e273e378"
server_key_iv: "8ca3c2b62189d76f5ea104d1bc13ab18"
server_key_version: "encryptionkey@0"
I20260929 15:45:00.862344 6097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 15:45:00.866643 6112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:00.868575 6097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:00.868781 6097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5a3904efb3514162a0447220f4110dc4"
format_stamp: "Formatted at 2026-09-29 15:45:00 on dist-test-slave-7491"
server_key: "8753ac073754a6d30b4374b8e273e378"
server_key_iv: "8ca3c2b62189d76f5ea104d1bc13ab18"
server_key_version: "encryptionkey@0"
I20260929 15:45:00.869016 6097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:00.885237 6097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:00.890519 6097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:00.890882 6097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:00.892009 6097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:00.893772 6097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:00.893858 6097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:00.893949 6097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:00.894001 6097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:00.934556 6097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42703
I20260929 15:45:00.934600 6224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42703 every 8 connection(s)
I20260929 15:45:00.936084 6097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:00.942830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6097
I20260929 15:45:00.943094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:00.943928 388 external_mini_cluster.cc:1428] Setting key ad79862d1d7e8cf921695e92c859c952
Sep 29 15:45:00 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:00.967221 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:00.939668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58475 (local address 127.0.97.62:46101)
0929 15:45:00.940238 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:00.940244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:00.941931 (+ 1687us) server_negotiation.cc:407] Connection header received
0929 15:45:00.944978 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:00.944984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:00.945092 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:00.945227 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:00.947530 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:00.948438 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:00.951429 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:00.952022 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:00.952297 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:00.958224 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:00.958232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:00.958236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:00.958289 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:00.961627 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:00.963221 (+ 1594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:00.963229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:00.963232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:00.963478 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:00.964415 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:00.964420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:00.964422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:00.964817 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:00.964970 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:00.966368 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0929 15:45:00.966664 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260929 15:45:00.968411 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:00.940591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46101 (local address 127.0.97.1:58475)
0929 15:45:00.941697 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:00.941764 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:00.944584 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:00.945556 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:00.945567 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:00.946268 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:00.947263 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:00.947296 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:00.948634 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:00.948644 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:00.951088 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:00.951113 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:00.952602 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:00.952621 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:00.953192 (+ 571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:00.954466 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:00.954496 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:00.957828 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:00.961943 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:00.961962 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:00.961994 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:00.962949 (+ 955us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:00.963715 (+ 766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:00.963724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:00.963731 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:00.964094 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:00.965023 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:00.965038 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:00.965726 (+ 688us) client_negotiation.cc:769] Sending connection context
0929 15:45:00.966204 (+ 478us) client_negotiation.cc:240] Negotiation successful
0929 15:45:00.966931 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":437,"threads_started":1}
I20260929 15:45:00.970425 6225 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46101
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:45:00.970794 6225 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:00.971621 6225 heartbeater.cc:511] Master 127.0.97.62:46101 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:45:00.974974 6036 ts_manager.cc:194] Registered new tserver with Master: 5a3904efb3514162a0447220f4110dc4 (127.0.97.1:42703)
I20260929 15:45:00.977032 6036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:45:01.004946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46101
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:01.411634 6231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:01.412022 6231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:01.412242 6231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:01.419924 6231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:01.420123 6231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:01.420341 6231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:01.430919 6231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:01.432606 6231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:01.434078 6231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:01.450030 6237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:01.450111 6236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:01.450139 6239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:01.451175 6231 server_base.cc:1034] running on GCE node
I20260929 15:45:01.452154 6231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:01.453747 6231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:01.455217 6231 hybrid_clock.cc:648] HybridClock initialized: now 1790696701455138 us; error 51 us; skew 500 ppm
I20260929 15:45:01.455597 6231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:01 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:01.460846 6231 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:45:01.462698 6231 webserver.cc:460] Webserver started at http://127.0.97.2:45161/ using document root <none> and password file <none>
I20260929 15:45:01.463539 6231 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:01.463714 6231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:01.464061 6231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:01.467409 6231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "7236f743751346fdb8096c99a9011ce4"
format_stamp: "Formatted at 2026-09-29 15:45:01 on dist-test-slave-7491"
server_key: "94e9a3ef17ea166654f0a2dcd56b363b"
server_key_iv: "c2a29e17e1b217a6d6188a57d7cf355c"
server_key_version: "encryptionkey@0"
I20260929 15:45:01.468458 6231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7236f743751346fdb8096c99a9011ce4"
format_stamp: "Formatted at 2026-09-29 15:45:01 on dist-test-slave-7491"
server_key: "94e9a3ef17ea166654f0a2dcd56b363b"
server_key_iv: "c2a29e17e1b217a6d6188a57d7cf355c"
server_key_version: "encryptionkey@0"
I20260929 15:45:01.475694 6231 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 15:45:01.480927 6246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:01.482923 6231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:45:01.483157 6231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "7236f743751346fdb8096c99a9011ce4"
format_stamp: "Formatted at 2026-09-29 15:45:01 on dist-test-slave-7491"
server_key: "94e9a3ef17ea166654f0a2dcd56b363b"
server_key_iv: "c2a29e17e1b217a6d6188a57d7cf355c"
server_key_version: "encryptionkey@0"
I20260929 15:45:01.483426 6231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:01.508843 6231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:01.514773 6231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:01.515378 6231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:01.517139 6231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:01.519140 6231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:01.519294 6231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:01.519465 6231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:01.519551 6231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:01.562772 6231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40353
I20260929 15:45:01.562880 6358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40353 every 8 connection(s)
I20260929 15:45:01.564302 6231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:01.573609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6231
I20260929 15:45:01.573835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:01.574325 388 external_mini_cluster.cc:1428] Setting key bec389c53dc03c4c7eda88f6ff411c11
Sep 29 15:45:01 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:01.588047 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:01.567291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41715 (local address 127.0.97.62:46101)
0929 15:45:01.567804 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:01.567809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:01.569150 (+ 1341us) server_negotiation.cc:407] Connection header received
0929 15:45:01.571219 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:01.571225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:01.571331 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:01.571470 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:01.573397 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:01.574359 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:01.576349 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:01.576833 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:01.576987 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:01.581554 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:01.581561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:01.581564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:01.581600 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:01.584553 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:01.585475 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:01.585481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:01.585484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:01.585615 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:01.586146 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:01.586155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:01.586157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:01.586467 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:01.586635 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:01.587395 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 15:45:01.587651 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
I20260929 15:45:01.589303 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:01.567727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46101 (local address 127.0.97.2:41715)
0929 15:45:01.568942 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:01.568988 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:01.570843 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:01.571755 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:01.571764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:01.572469 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:01.573112 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:01.573133 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:01.574575 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:01.574581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:01.576097 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:01.576112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:01.577097 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:01.577105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:01.577374 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:01.578258 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:01.578286 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:01.581219 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:01.584748 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:01.584760 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:01.584781 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:01.585290 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:01.585737 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:01.585742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:01.585746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:01.586018 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:01.586606 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:01.586615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:01.587009 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 15:45:01.587301 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 15:45:01.587906 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":509,"threads_started":1}
I20260929 15:45:01.591185 6359 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46101
I20260929 15:45:01.591634 6359 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:01.592744 6359 heartbeater.cc:511] Master 127.0.97.62:46101 requested a full tablet report, sending...
I20260929 15:45:01.595213 6036 ts_manager.cc:194] Registered new tserver with Master: 7236f743751346fdb8096c99a9011ce4 (127.0.97.2:40353)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:45:01.596633 6036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:45:01.629483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46101
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:01.981576 6225 heartbeater.cc:503] Master 127.0.97.62:46101 was elected leader, sending a full tablet report...
W20260929 15:45:02.000866 6365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:02.001215 6365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:02.001315 6365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:02.009541 6365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:02.009701 6365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:02.009821 6365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:02.020740 6365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:02.022202 6365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:02.023589 6365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:02.039789 6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:02.039883 6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:02.040330 6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:02.042380 6365 server_base.cc:1034] running on GCE node
I20260929 15:45:02.043128 6365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:02.044281 6365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:02.045620 6365 hybrid_clock.cc:648] HybridClock initialized: now 1790696702045541 us; error 41 us; skew 500 ppm
I20260929 15:45:02.046048 6365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.049722 6365 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:45:02.051584 6365 webserver.cc:460] Webserver started at http://127.0.97.3:37737/ using document root <none> and password file <none>
I20260929 15:45:02.052497 6365 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:02.052623 6365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:02.052906 6365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:02.056138 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "a8bd74666c114a7d9073de7724524f12"
format_stamp: "Formatted at 2026-09-29 15:45:02 on dist-test-slave-7491"
server_key: "fb7855f4f0863d2ded8f82f5031633b7"
server_key_iv: "8f7fe720d14495da735c5dc9718d1438"
server_key_version: "encryptionkey@0"
I20260929 15:45:02.057085 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a8bd74666c114a7d9073de7724524f12"
format_stamp: "Formatted at 2026-09-29 15:45:02 on dist-test-slave-7491"
server_key: "fb7855f4f0863d2ded8f82f5031633b7"
server_key_iv: "8f7fe720d14495da735c5dc9718d1438"
server_key_version: "encryptionkey@0"
I20260929 15:45:02.062892 6365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 15:45:02.067116 6380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:02.069612 6365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:02.069906 6365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "a8bd74666c114a7d9073de7724524f12"
format_stamp: "Formatted at 2026-09-29 15:45:02 on dist-test-slave-7491"
server_key: "fb7855f4f0863d2ded8f82f5031633b7"
server_key_iv: "8f7fe720d14495da735c5dc9718d1438"
server_key_version: "encryptionkey@0"
I20260929 15:45:02.070178 6365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:02.099697 6365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:02.104993 6365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:02.105320 6365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:02.106449 6365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:02.108098 6365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:02.108247 6365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 15:45:02.108412 6365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:02.108486 6365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:02.148102 6365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34047
I20260929 15:45:02.148118 6492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34047 every 8 connection(s)
I20260929 15:45:02.149725 6365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:02.158525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6365
I20260929 15:45:02.158725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:02.159235 388 external_mini_cluster.cc:1428] Setting key d1527fdedaac1707c7a5a8df293c199d
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:02.176501 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.152826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40241 (local address 127.0.97.62:46101)
0929 15:45:02.153341 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.153347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.155058 (+ 1711us) server_negotiation.cc:407] Connection header received
0929 15:45:02.157174 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.157180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.157291 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.157406 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.159705 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.160716 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.163440 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.164306 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.164444 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.168808 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.168815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.168818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.168864 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.171847 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.172897 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.172903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.172907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.173048 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.173549 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.173558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.173560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.173902 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.174112 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.175597 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.176041 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":244,"threads_started":1}
I20260929 15:45:02.177729 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.153565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46101 (local address 127.0.97.3:40241)
0929 15:45:02.154797 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:02.154854 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:02.156779 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:02.157815 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:02.157826 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:02.158467 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:02.159311 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:02.159341 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.161008 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.161015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:02.163164 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:02.163191 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.164658 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.164666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:02.164956 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.165818 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:02.165845 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:02.168492 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:02.172082 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:02.172102 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:02.172120 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:02.172734 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:02.173180 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:02.173185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:02.173190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:02.173412 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:02.174071 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:02.174081 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:02.174622 (+ 541us) client_negotiation.cc:769] Sending connection context
0929 15:45:02.175366 (+ 744us) client_negotiation.cc:240] Negotiation successful
0929 15:45:02.176255 (+ 889us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":712,"threads_started":1}
I20260929 15:45:02.179746 6493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46101
I20260929 15:45:02.180226 6493 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:02.181120 6493 heartbeater.cc:511] Master 127.0.97.62:46101 requested a full tablet report, sending...
I20260929 15:45:02.182870 6036 ts_manager.cc:194] Registered new tserver with Master: a8bd74666c114a7d9073de7724524f12 (127.0.97.3:34047)
I20260929 15:45:02.183743 6036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40241
I20260929 15:45:02.190905 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/test-admin.keytab.
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.223228 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 15:45:02.230226 6501 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:02.226532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.0.97.1:42703)
0929 15:45:02.227778 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.227786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.227819 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:45:02.227897 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.227904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.228147 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.228406 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.229600 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51756: BlockingRecv error: recv got EOF from 127.0.0.1:51756 (error 108)
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":713,"threads_started":1}
W20260929 15:45:02.230259 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.232734 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.235005 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:45:02.254791 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.237205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.0.97.1:42703)
0929 15:45:02.237494 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.237502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.237823 (+ 321us) server_negotiation.cc:407] Connection header received
0929 15:45:02.237995 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.237999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.238081 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.238179 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.239825 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.240974 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.242650 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.243437 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.243591 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.248293 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.248312 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.248330 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.248396 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.251511 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.252461 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.252470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.252472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.252616 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.253030 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.253036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.253040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.253473 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.253796 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.254167 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.254478 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260929 15:45:02.263067 6501 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:02.260706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.0.97.1:42703)
0929 15:45:02.260963 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.260969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.261032 (+ 63us) server_negotiation.cc:407] Connection header received
0929 15:45:02.261389 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.261394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.261489 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.261610 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.262830 (+ 1220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51776: BlockingRecv error: recv got EOF from 127.0.0.1:51776 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 15:45:02.263478 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.265430 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.267009 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:45:02.288905 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.268837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.0.97.1:42703)
0929 15:45:02.269105 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.269111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.269334 (+ 223us) server_negotiation.cc:407] Connection header received
0929 15:45:02.269877 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.269882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.269981 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.270116 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.271452 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.272370 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.274029 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.274688 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.274837 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.280747 (+ 5910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.280755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.280760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.280845 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.284139 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.285532 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.285539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.285542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.285863 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.286611 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.286618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.286621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.287082 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.287344 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.287989 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.288344 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 15:45:02.294402 6501 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:02.292408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.97.1:42703)
0929 15:45:02.292652 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.292657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.292778 (+ 121us) server_negotiation.cc:407] Connection header received
0929 15:45:02.292964 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.292969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.293068 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.293169 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.294147 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51792: BlockingRecv error: recv got EOF from 127.0.0.1:51792 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260929 15:45:02.294577 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790696645767654-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.296939 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:02.298542 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:45:02.315972 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.300480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.0.97.1:42703)
0929 15:45:02.300758 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.300766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.301191 (+ 425us) server_negotiation.cc:407] Connection header received
0929 15:45:02.301430 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.301436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.301510 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.301621 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.302712 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.303542 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.304945 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.305382 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.305529 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.309648 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.309655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.309658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.309828 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.313104 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.313929 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.313934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.313936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.314101 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.314489 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.314499 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.314501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.314786 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.315121 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.315394 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.315626 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:45:02.318217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6097
I20260929 15:45:02.334208 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.319391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.97.1:42703)
0929 15:45:02.319681 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.319691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.320037 (+ 346us) server_negotiation.cc:407] Connection header received
0929 15:45:02.320438 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.320443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.320551 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.320689 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.322006 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.322739 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.324090 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.324660 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.324971 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.327461 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.327467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.327471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.327516 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.330921 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.331848 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.331854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.331858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.332007 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.332562 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.332566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.332568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.332937 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.333173 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.333573 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.333865 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:45:02.335224 6220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:02.431365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6097
I20260929 15:45:02.452836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6231
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:45:02.472458 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.453848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.0.97.2:40353)
0929 15:45:02.454344 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.454351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.454383 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:45:02.454446 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.454453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.454668 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.454845 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.456511 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.457939 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.459458 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.460072 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.460284 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.464710 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.464735 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.464751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.464802 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.468665 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.469781 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.469793 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.469795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.470011 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.470506 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.470510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.470512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.470912 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.471173 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.471696 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.472062 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
I20260929 15:45:02.473898 6354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:02.566859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6231
I20260929 15:45:02.586123 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6365
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:45:02.608975 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:02.587410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.0.97.3:34047)
0929 15:45:02.588278 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:02.588294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:02.588352 (+ 58us) server_negotiation.cc:407] Connection header received
0929 15:45:02.588444 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:02.588452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:02.588819 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:02.589026 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:02.590477 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.591547 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.593781 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:02.594728 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:02.594893 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:02.601011 (+ 6118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:02.601034 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:02.601056 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:02.601119 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:02.605102 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.606067 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.606077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.606088 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.606249 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:02.606749 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:02.606753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:02.606755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:02.607160 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:02.607496 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:02.608080 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 15:45:02.608555 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":361,"threads_started":1}
I20260929 15:45:02.610744 6488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:02.707955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6365
I20260929 15:45:02.727182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6004
I20260929 15:45:02.728464 6066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:02.845991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6004
2026-09-29T15:45:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4706 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[6521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[6521](info): set up 2 sockets
Sep 29 15:45:02 dist-test-slave-7491 krb5kdc[6521](info): commencing operation
krb5kdc: starting...
W20260929 15:45:04.017119 388 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:45:04 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 15:45:04.272554 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:04.664546 6534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:04.664893 6534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:04.665016 6534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:04.672992 6534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:04.673146 6534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:04.673219 6534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:04.673336 6534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:04.673409 6534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:04.685992 6534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:04.688448 6534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:04.691063 6534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:04.706925 6540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:04.706993 6542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:04.707083 6539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:04.707736 6534 server_base.cc:1034] running on GCE node
I20260929 15:45:04.708755 6534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:04.710479 6534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:04.711796 6534 hybrid_clock.cc:648] HybridClock initialized: now 1790696704711728 us; error 55 us; skew 500 ppm
I20260929 15:45:04.712237 6534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:04 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:04.717083 6534 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 15:45:04.719125 6534 webserver.cc:460] Webserver started at http://127.0.97.62:46853/ using document root <none> and password file <none>
I20260929 15:45:04.720268 6534 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:04.720449 6534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:04.720836 6534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:04.724745 6534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "bdfc10a7ac2c4cfebb70413ae3961488"
format_stamp: "Formatted at 2026-09-29 15:45:04 on dist-test-slave-7491"
server_key: "8ebe427251a21ba8294589d9cb60db47"
server_key_iv: "15b57e6b99f4ed1420c56de995f9e719"
server_key_version: "encryptionkey@0"
I20260929 15:45:04.725662 6534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "bdfc10a7ac2c4cfebb70413ae3961488"
format_stamp: "Formatted at 2026-09-29 15:45:04 on dist-test-slave-7491"
server_key: "8ebe427251a21ba8294589d9cb60db47"
server_key_iv: "15b57e6b99f4ed1420c56de995f9e719"
server_key_version: "encryptionkey@0"
I20260929 15:45:04.731750 6534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:45:04.737385 6549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:04.739322 6534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:45:04.739601 6534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "bdfc10a7ac2c4cfebb70413ae3961488"
format_stamp: "Formatted at 2026-09-29 15:45:04 on dist-test-slave-7491"
server_key: "8ebe427251a21ba8294589d9cb60db47"
server_key_iv: "15b57e6b99f4ed1420c56de995f9e719"
server_key_version: "encryptionkey@0"
I20260929 15:45:04.739863 6534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:04.771080 6534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:04.776516 6534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:04.776866 6534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:04.800962 6534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38733
I20260929 15:45:04.800967 6600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38733 every 8 connection(s)
I20260929 15:45:04.802577 6534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:04.807751 6601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:04.811803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6534
I20260929 15:45:04.812083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:04.812736 388 external_mini_cluster.cc:1428] Setting key a49468587b883182036fa3f3e14af16d
I20260929 15:45:04.822181 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Bootstrap starting.
I20260929 15:45:04.826563 6601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:04.827853 6601 log.cc:824] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:45:04 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:04.831854 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: No bootstrap required, opened a new log
I20260929 15:45:04.840570 6601 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260929 15:45:04.841178 6601 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:04.841353 6601 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfc10a7ac2c4cfebb70413ae3961488, State: Initialized, Role: FOLLOWER
I20260929 15:45:04.842336 6601 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260929 15:45:04.842723 6601 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:04.842921 6601 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:04.842944 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:04.815336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.97.62:38733)
0929 15:45:04.816420 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:04.816440 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:04.816498 (+ 58us) server_negotiation.cc:407] Connection header received
0929 15:45:04.819512 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:04.819555 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:04.820154 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:04.820751 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:04.823549 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:04.825095 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:04.827007 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:04.827814 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:04.827971 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:04.833591 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:04.833604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:04.833624 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:04.833675 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:04.837254 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:04.838128 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:04.838139 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:04.838148 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:04.838349 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:04.838950 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:04.838963 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:04.838966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:04.839802 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:04.840147 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:04.840811 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 15:45:04.841652 (+ 841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":414,"threads_started":1}
I20260929 15:45:04.843205 6601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:04.846774 6601 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260929 15:45:04.847450 6601 leader_election.cc:302] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdfc10a7ac2c4cfebb70413ae3961488; no voters:
I20260929 15:45:04.848119 6601 leader_election.cc:288] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:04.848369 6606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:04.849056 6606 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 LEADER]: Becoming Leader. State: Replica: bdfc10a7ac2c4cfebb70413ae3961488, State: Running, Role: LEADER
I20260929 15:45:04.849637 6606 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } }
I20260929 15:45:04.850777 6601 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:04.853608 6608 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdfc10a7ac2c4cfebb70413ae3961488. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } }
I20260929 15:45:04.854393 6608 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:04.855050 6607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } }
I20260929 15:45:04.855443 6607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:04.856901 6613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:04.864221 6613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:04.875288 6613 catalog_manager.cc:1347] Generated new cluster ID: 1e8830dd33ae4aafac80e4f49874a95c
I20260929 15:45:04.875447 6613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:04.896966 6613 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:04.899148 6613 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:04.906903 6613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Generated new TSK 0
I20260929 15:45:04.907881 6613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 15:45:04.977077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:05.428470 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:05.428804 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:05.428906 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:05.437575 6627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:05.437728 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:05.437858 6627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:05.449515 6627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:05.451934 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:05.453637 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:05.469447 6632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:05.469684 6635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:05.469709 6633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:05.470683 6627 server_base.cc:1034] running on GCE node
I20260929 15:45:05.471359 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:05.472469 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:05.473721 6627 hybrid_clock.cc:648] HybridClock initialized: now 1790696705473640 us; error 72 us; skew 500 ppm
I20260929 15:45:05.474153 6627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:05 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:05.482470 6627 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:45:05.484730 6627 webserver.cc:460] Webserver started at http://127.0.97.1:34461/ using document root <none> and password file <none>
I20260929 15:45:05.485623 6627 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:05.485795 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:05.486143 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:05.489555 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "77332a715cbc4e62ac3edf365ad3b939"
format_stamp: "Formatted at 2026-09-29 15:45:05 on dist-test-slave-7491"
server_key: "7806db3083791f855b0db65059e7e0eb"
server_key_iv: "9ee4e58a8fe9c0cb54d8f7076614830c"
server_key_version: "encryptionkey@0"
I20260929 15:45:05.490577 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "77332a715cbc4e62ac3edf365ad3b939"
format_stamp: "Formatted at 2026-09-29 15:45:05 on dist-test-slave-7491"
server_key: "7806db3083791f855b0db65059e7e0eb"
server_key_iv: "9ee4e58a8fe9c0cb54d8f7076614830c"
server_key_version: "encryptionkey@0"
I20260929 15:45:05.498353 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 15:45:05.502803 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:05.504921 6627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:45:05.505173 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "77332a715cbc4e62ac3edf365ad3b939"
format_stamp: "Formatted at 2026-09-29 15:45:05 on dist-test-slave-7491"
server_key: "7806db3083791f855b0db65059e7e0eb"
server_key_iv: "9ee4e58a8fe9c0cb54d8f7076614830c"
server_key_version: "encryptionkey@0"
I20260929 15:45:05.505426 6627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:05.527221 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:05.533861 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:05.534267 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:05.535561 6627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:05.537753 6627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:05.537890 6627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:05.538060 6627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:05.538228 6627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:05.580600 6627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45041
I20260929 15:45:05.580613 6754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45041 every 8 connection(s)
I20260929 15:45:05.582450 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:05.583570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6627
I20260929 15:45:05.583913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:05.584875 388 external_mini_cluster.cc:1428] Setting key 522cf11aa95335af71279c7a73cdcac1
Sep 29 15:45:05 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:45:05.612735 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:05.586293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47393 (local address 127.0.97.62:38733)
0929 15:45:05.586822 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:05.586830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:05.588493 (+ 1663us) server_negotiation.cc:407] Connection header received
0929 15:45:05.590899 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:05.590909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:05.591020 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:05.591198 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:05.593372 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:05.594736 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:05.597776 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:05.598497 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:05.598663 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:05.604331 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:05.604338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:05.604341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:05.604385 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:05.607745 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:05.609188 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:05.609194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:05.609197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:05.609370 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:05.610130 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:05.610134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:05.610136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:05.610471 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:05.610655 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:05.611990 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0929 15:45:05.612370 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
I20260929 15:45:05.613891 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:05.587044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:47393)
0929 15:45:05.588295 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:05.588359 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:05.590472 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:05.591533 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:05.591544 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:05.592394 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:05.593077 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:05.593101 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:05.595036 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:05.595045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:05.597519 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:05.597546 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:05.598824 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:05.598835 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:05.599224 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:05.600247 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:05.600291 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:05.603843 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:05.608010 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:05.608028 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:05.608058 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:05.608980 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:05.609606 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:05.609614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:05.609621 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:05.609960 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:05.610638 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:05.610658 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:05.611240 (+ 582us) client_negotiation.cc:769] Sending connection context
0929 15:45:05.611631 (+ 391us) client_negotiation.cc:240] Negotiation successful
0929 15:45:05.612387 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":492,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260929 15:45:05.615854 6755 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260929 15:45:05.616263 6755 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:05.617110 6755 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260929 15:45:05.620048 6566 ts_manager.cc:194] Registered new tserver with Master: 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041)
I20260929 15:45:05.623041 6566 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 15:45:05.644562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:06.069589 6761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:06.069922 6761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:06.070020 6761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:06.078945 6761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:06.079092 6761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:06.079223 6761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:06.090221 6761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:06.091665 6761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:06.093340 6761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:06.108522 6766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:06.108579 6767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:06.108531 6769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:06.109869 6761 server_base.cc:1034] running on GCE node
I20260929 15:45:06.110900 6761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:06.112361 6761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:06.113732 6761 hybrid_clock.cc:648] HybridClock initialized: now 1790696706113609 us; error 85 us; skew 500 ppm
I20260929 15:45:06.114188 6761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:06 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:06.117995 6761 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:45:06.120410 6761 webserver.cc:460] Webserver started at http://127.0.97.2:39415/ using document root <none> and password file <none>
I20260929 15:45:06.121490 6761 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:06.121672 6761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:06.122035 6761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:06.125466 6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "09a0b7a360cb482e8388398c020630c8"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "ed1b0dedb3a4ef860491369f1d2a33ca"
server_key_iv: "4ebb3b63423a4e5bcac9cfb65c14014d"
server_key_version: "encryptionkey@0"
I20260929 15:45:06.126396 6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "09a0b7a360cb482e8388398c020630c8"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "ed1b0dedb3a4ef860491369f1d2a33ca"
server_key_iv: "4ebb3b63423a4e5bcac9cfb65c14014d"
server_key_version: "encryptionkey@0"
I20260929 15:45:06.133033 6761 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:45:06.137820 6776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:06.139917 6761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:45:06.140293 6761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "09a0b7a360cb482e8388398c020630c8"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "ed1b0dedb3a4ef860491369f1d2a33ca"
server_key_iv: "4ebb3b63423a4e5bcac9cfb65c14014d"
server_key_version: "encryptionkey@0"
I20260929 15:45:06.140592 6761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:06.166102 6761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:06.171367 6761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:06.171782 6761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:06.173033 6761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:06.174674 6761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:06.175026 6761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:06.175395 6761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:06.175501 6761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:06.217404 6761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45411
I20260929 15:45:06.217525 6888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45411 every 8 connection(s)
I20260929 15:45:06.218887 6761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:06.222510 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6761
I20260929 15:45:06.222807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:06.223419 388 external_mini_cluster.cc:1428] Setting key c73127c7998ec5ac2ebb1cb5370019e0
Sep 29 15:45:06 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:06.249944 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:06.222962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36553 (local address 127.0.97.62:38733)
0929 15:45:06.223540 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:06.223547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:06.224900 (+ 1353us) server_negotiation.cc:407] Connection header received
0929 15:45:06.227680 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:06.227685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:06.227782 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:06.227938 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:06.230602 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.231466 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.234278 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.235195 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.235387 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:06.240054 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:06.240064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:06.240073 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:06.240132 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:06.245389 (+ 5257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:06.246661 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:06.246668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:06.246671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:06.246853 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:06.247518 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:06.247522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:06.247525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:06.248005 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:06.248221 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:06.249410 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0929 15:45:06.249634 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":263,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:45:06.251533 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:06.223646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:36553)
0929 15:45:06.224630 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:06.224699 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:06.227203 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:06.228340 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:06.228382 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:06.229283 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:06.230287 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:06.230331 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.231704 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.231713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:06.233998 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:06.234017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.235493 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.235498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:06.235742 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:06.236620 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:06.236657 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:06.239615 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:06.245703 (+ 6088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:06.245717 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:06.245742 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:06.246473 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:06.247001 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:06.247006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:06.247011 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:06.247344 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:06.248250 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:06.248261 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:06.248842 (+ 581us) client_negotiation.cc:769] Sending connection context
0929 15:45:06.249221 (+ 379us) client_negotiation.cc:240] Negotiation successful
0929 15:45:06.249966 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":349,"threads_started":1}
I20260929 15:45:06.253340 6889 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260929 15:45:06.253777 6889 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:06.254616 6889 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260929 15:45:06.256815 6566 ts_manager.cc:194] Registered new tserver with Master: 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411)
I20260929 15:45:06.257730 6566 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 15:45:06.284510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:06.626760 6755 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
W20260929 15:45:06.688984 6895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:06.689317 6895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:06.689494 6895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:06.697949 6895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:06.698104 6895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:06.698299 6895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:06.712029 6895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:06.713675 6895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:06.715481 6895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:06.730988 6900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:06.732498 6901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:06.733516 6903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:06.733889 6895 server_base.cc:1034] running on GCE node
I20260929 15:45:06.734623 6895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:06.735849 6895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:06.737254 6895 hybrid_clock.cc:648] HybridClock initialized: now 1790696706737187 us; error 83 us; skew 500 ppm
I20260929 15:45:06.737771 6895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:06 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:06.741910 6895 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:45:06.743909 6895 webserver.cc:460] Webserver started at http://127.0.97.3:39211/ using document root <none> and password file <none>
I20260929 15:45:06.744922 6895 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:06.745096 6895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:06.745460 6895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:06.749187 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "3c17202443ee4dcdb74528f25fd6578f"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "4febf9d037a0fbbe4983ddd85997c97a"
server_key_iv: "0a2f2057c890e33e72ce6e501429afa2"
server_key_version: "encryptionkey@0"
I20260929 15:45:06.750208 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3c17202443ee4dcdb74528f25fd6578f"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "4febf9d037a0fbbe4983ddd85997c97a"
server_key_iv: "0a2f2057c890e33e72ce6e501429afa2"
server_key_version: "encryptionkey@0"
I20260929 15:45:06.758423 6895 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 15:45:06.763072 6910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:06.765092 6895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:06.765336 6895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "3c17202443ee4dcdb74528f25fd6578f"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "4febf9d037a0fbbe4983ddd85997c97a"
server_key_iv: "0a2f2057c890e33e72ce6e501429afa2"
server_key_version: "encryptionkey@0"
I20260929 15:45:06.765591 6895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:06.788038 6895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:06.794214 6895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:06.794621 6895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:06.795753 6895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:06.797482 6895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:06.797595 6895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:06.797771 6895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:06.797850 6895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:06.840588 6895 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34081
I20260929 15:45:06.840718 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34081 every 8 connection(s)
I20260929 15:45:06.842103 6895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:06.844929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 6895
I20260929 15:45:06.845270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:06.845894 388 external_mini_cluster.cc:1428] Setting key 65c1d3fa1d8ad19463a9f7f273bde350
Sep 29 15:45:06 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:06.869405 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:06.845590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49123 (local address 127.0.97.62:38733)
0929 15:45:06.846263 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:06.846271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:06.847512 (+ 1241us) server_negotiation.cc:407] Connection header received
0929 15:45:06.849732 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:06.849737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:06.849840 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:06.849962 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:06.852239 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.853049 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.855657 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.856156 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.856384 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:06.861535 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:06.861542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:06.861547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:06.861604 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:06.865413 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:06.866476 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:06.866482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:06.866484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:06.866629 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:06.867230 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:06.867235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:06.867237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:06.867551 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:06.867674 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:06.868735 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0929 15:45:06.869030 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":300,"threads_started":1}
I20260929 15:45:06.870487 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:06.846355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:49123)
0929 15:45:06.847356 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:06.847412 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:06.849375 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:06.850243 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:06.850252 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:06.851128 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:06.851968 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:06.851996 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.853235 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.853241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:06.855305 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:06.855333 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:06.856551 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:06.856560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:06.856912 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:06.857837 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:06.857867 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:06.861089 (+ 3222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:06.865656 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:06.865670 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:06.865692 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:06.866287 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:06.866878 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:06.866882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:06.866884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:06.867098 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:06.867750 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:06.867757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:06.868154 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 15:45:06.868551 (+ 397us) client_negotiation.cc:240] Negotiation successful
0929 15:45:06.869119 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":401,"threads_started":1}
I20260929 15:45:06.873204 7023 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260929 15:45:06.873713 7023 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:06.874835 7023 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260929 15:45:06.876997 6565 ts_manager.cc:194] Registered new tserver with Master: 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081)
I20260929 15:45:06.878052 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49123
I20260929 15:45:06.890642 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 15:45:06.948915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38801
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38733,127.0.97.61:38801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:07.261374 6889 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
W20260929 15:45:07.336058 7029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:07.336476 7029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:07.336611 7029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:07.344741 7029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:07.344872 7029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:07.344920 7029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:07.345000 7029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:07.345068 7029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:07.356079 7029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38733,127.0.97.61:38801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
W20260929 15:45:07.356563 7029 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:38733,127.0.97.61:38801'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 15:45:07.357673 7029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:07.359103 7029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:07.373047 7037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:07.373098 7035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:07.373013 7034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:07.380033 7029 server_base.cc:1034] running on GCE node
I20260929 15:45:07.381274 7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:07.382963 7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:07.384506 7029 hybrid_clock.cc:648] HybridClock initialized: now 1790696707384376 us; error 97 us; skew 500 ppm
I20260929 15:45:07.384929 7029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:07 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:07.388706 7029 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 15:45:07.390473 7029 webserver.cc:460] Webserver started at http://127.0.97.61:34613/ using document root <none> and password file <none>
I20260929 15:45:07.391235 7029 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:07.391430 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:07.391805 7029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:07.396248 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/instance:
uuid: "115035d166124ced826936fab2b13803"
format_stamp: "Formatted at 2026-09-29 15:45:07 on dist-test-slave-7491"
server_key: "f61ea075984120dd536812cf0596569a"
server_key_iv: "6937184c4604fbff113be453e9925434"
server_key_version: "encryptionkey@0"
I20260929 15:45:07.397996 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal/instance:
uuid: "115035d166124ced826936fab2b13803"
format_stamp: "Formatted at 2026-09-29 15:45:07 on dist-test-slave-7491"
server_key: "f61ea075984120dd536812cf0596569a"
server_key_iv: "6937184c4604fbff113be453e9925434"
server_key_version: "encryptionkey@0"
I20260929 15:45:07.405455 7029 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 15:45:07.410051 7044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:07.411690 7029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 15:45:07.412014 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
uuid: "115035d166124ced826936fab2b13803"
format_stamp: "Formatted at 2026-09-29 15:45:07 on dist-test-slave-7491"
server_key: "f61ea075984120dd536812cf0596569a"
server_key_iv: "6937184c4604fbff113be453e9925434"
server_key_version: "encryptionkey@0"
I20260929 15:45:07.412386 7029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:07.431272 7029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:07.438050 7029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:07.438423 7029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:07.465358 7029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38801
I20260929 15:45:07.465440 7095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38801 every 8 connection(s)
I20260929 15:45:07.467235 7029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:45:07.472586 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:07.476059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 7029
I20260929 15:45:07.482681 7096 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:45:07 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 15:45:07 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:07.507609 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.486670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.97.62:38733)
0929 15:45:07.488088 (+ 1418us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.488095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.488115 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:45:07.488216 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.488221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.488296 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.488420 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.490495 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.491733 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.494226 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.494672 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.494797 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.499975 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.499993 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.500000 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.500081 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.503290 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.504726 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.504732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.504743 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.504997 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.505711 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.505716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.505718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.506108 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.506300 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.506929 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.507203 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":1202,"threads_started":1}
I20260929 15:45:07.509028 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.479273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.0.97.61:38801)
0929 15:45:07.480715 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.480750 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.480819 (+ 69us) server_negotiation.cc:407] Connection header received
0929 15:45:07.483010 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.483041 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.483650 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.484241 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.486328 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.488674 (+ 2346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.490911 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.492084 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.492328 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.499002 (+ 6674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.499018 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.499038 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.499084 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.503066 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.504489 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.504500 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.504504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.504776 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.505330 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.505335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.505338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.506119 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.506452 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.507018 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.507354 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":764,"threads_started":1}
I20260929 15:45:07.509052 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.486519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52330)
0929 15:45:07.487499 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:07.487568 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:07.487834 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:07.488629 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:07.488637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:07.489887 (+ 1250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:07.490267 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.490281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.491930 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.491934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.493995 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.494014 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.495101 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.495114 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.495423 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.496825 (+ 1402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:07.496877 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:07.499677 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:07.503541 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.503558 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.503572 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.504501 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.505165 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.505169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.505172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.505463 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.506274 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:07.506282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:07.506525 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 15:45:07.506778 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 15:45:07.507499 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":188,"thread_start_us":359,"threads_started":1}
I20260929 15:45:07.513006 7096 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:45:07 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:45:07.529101 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.513864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39598)
0929 15:45:07.514169 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:07.514194 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:07.514467 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:07.514870 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:07.514874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:07.515285 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:07.515417 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.515426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.517036 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.517045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.518079 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.518090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.518841 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.518846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.519221 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.520112 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:07.520132 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:07.522995 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:07.526495 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.526501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.526515 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.526978 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.527453 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.527457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.527459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.527595 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.528280 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:07.528283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:07.528427 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 15:45:07.528564 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 15:45:07.528823 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 15:45:07.529464 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.514238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.0.97.61:38801)
0929 15:45:07.514558 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.514566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.514588 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:45:07.514647 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.514653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.514742 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.514848 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.515557 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.516858 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.518247 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.518685 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.518829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.523264 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.523271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.523277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.523312 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.526283 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.527147 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.527153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.527155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.527289 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.527778 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.527782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.527784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.528088 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.528303 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.528701 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.528956 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:45:07.535683 7096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Bootstrap starting.
I20260929 15:45:07.540556 7096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:07.541666 7096 log.cc:824] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:07.544904 7096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: No bootstrap required, opened a new log
I20260929 15:45:07.552690 7096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } }
I20260929 15:45:07.553082 7096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:07.553201 7096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115035d166124ced826936fab2b13803, State: Initialized, Role: FOLLOWER
I20260929 15:45:07.553956 7096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } }
I20260929 15:45:07.555259 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } } }
I20260929 15:45:07.555680 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:45:07.556686 7096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:45:07.569882 7114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:45:07.570189 7114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:45:07.572276 7029 master_runner.cc:428] Detected that this master 115035d166124ced826936fab2b13803 is joining an existing cluster
I20260929 15:45:07.572563 7029 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38801
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260929 15:45:07.618889 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.579004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.0.97.62:38733)
0929 15:45:07.579341 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.579351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.596007 (+ 16656us) server_negotiation.cc:407] Connection header received
0929 15:45:07.596255 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.596262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.596386 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.596521 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.598685 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.600087 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.603411 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.604225 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.604384 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.609897 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.609905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.609910 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.609964 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.614864 (+ 4900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.615929 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.615935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.615939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.616107 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.616631 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.616637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.616641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.617135 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.617259 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.617975 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.618255 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260929 15:45:07.619666 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.579680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.0.97.61:38801)
0929 15:45:07.579877 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.579883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.591402 (+ 11519us) server_negotiation.cc:407] Connection header received
0929 15:45:07.591738 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.591745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.591817 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.591894 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.596232 (+ 4338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.597219 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.599296 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.600145 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.600318 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.609882 (+ 9564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.609893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.609900 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.609949 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.615295 (+ 5346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.616391 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.616399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.616405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.616599 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.617058 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.617065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.617071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.617503 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.617675 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.618232 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.618546 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260929 15:45:07.620591 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.579873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52338)
0929 15:45:07.595828 (+ 15955us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:07.595853 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:07.596042 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:07.596744 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:07.596795 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:07.597993 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:07.598395 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.598408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.600778 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.600785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.603159 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.603182 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.604603 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.604630 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.604948 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.607816 (+ 2868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:07.607866 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:07.609555 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:07.615115 (+ 5560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.615122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.615126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.615677 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.616298 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.616303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.616307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.616494 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.617355 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:07.617360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:07.617616 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 15:45:07.619344 (+ 1728us) client_negotiation.cc:240] Negotiation successful
0929 15:45:07.619851 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15754,"mutex_wait_us":36,"spinlock_wait_cycles":2816,"thread_start_us":10994,"threads_started":1}
I20260929 15:45:07.621986 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.579115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39614)
0929 15:45:07.591236 (+ 12121us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:07.591270 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:07.591558 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:07.592118 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:07.592122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:07.593806 (+ 1684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:07.594976 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.595015 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.597416 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.597422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.598785 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.598800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.600487 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.600505 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.600741 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.607779 (+ 7038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:07.607816 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:07.609603 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:07.615507 (+ 5904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.615515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.615522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.616239 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.616754 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.616761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.616768 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.616942 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.617766 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:07.617773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:07.617941 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 15:45:07.620243 (+ 2302us) client_negotiation.cc:240] Negotiation successful
0929 15:45:07.621415 (+ 1172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11951,"thread_start_us":16195,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260929 15:45:07.640321 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40467
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:45:07 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:45:07.648327 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.629302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39616)
0929 15:45:07.630205 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:07.630255 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:07.630444 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:07.630967 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:07.630975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:07.631876 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:07.632107 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.632116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.634221 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.634226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.635430 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.635451 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.636848 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.636855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.637134 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.637978 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:07.638145 (+ 167us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:07.641643 (+ 3498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:07.645379 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.645398 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.645405 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.645887 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.646309 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.646315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.646319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.646573 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.647203 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:07.647213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:07.647416 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 15:45:07.647622 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 15:45:07.647923 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":494,"threads_started":1}
I20260929 15:45:07.648991 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.629335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.0.97.61:38801)
0929 15:45:07.629629 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.629639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.630357 (+ 718us) server_negotiation.cc:407] Connection header received
0929 15:45:07.630698 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.630706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.630799 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.630917 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.632427 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.633956 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.635610 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.636560 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.636732 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.641933 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.641939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.641942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.641983 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.645158 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.646038 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.646051 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.646054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.646183 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.646704 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.646709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.646711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.647038 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.647215 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.648374 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.648654 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 15:45:07.651265 6565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38801: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "115035d166124ced826936fab2b13803"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38801
}
attrs {
promote: true
}
}
dest_uuid: "bdfc10a7ac2c4cfebb70413ae3961488"
cas_config_opid_index: -1
I20260929 15:45:07.652786 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } }
I20260929 15:45:07.668097 7071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:07.673885 7071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Refusing update from remote peer bdfc10a7ac2c4cfebb70413ae3961488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 15:45:07.675563 7128 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 115035d166124ced826936fab2b13803 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } }
I20260929 15:45:07.680805 7129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:07.681798 7128 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38801
I20260929 15:45:07.682807 7029 master.cc:537] Master@127.0.97.61:38801 shutting down...
W20260929 15:45:07.688066 6552 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 15:45:07.692497 7050 tablet.cc:2369] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:07.694131 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } }
I20260929 15:45:07.696455 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:07.703377 7029 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:07.703912 7029 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:07.704130 7029 tablet_replica.cc:330] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: stopping tablet replica
W20260929 15:45:07.706849 6552 consensus_peers.cc:487] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 -> Peer 115035d166124ced826936fab2b13803 (127.0.97.61:38801): Couldn't send request to peer 115035d166124ced826936fab2b13803. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 15:45:07.712476 7029 master.cc:552] Master@127.0.97.61:38801 shutdown complete.
I20260929 15:45:07.712625 7029 master_runner.cc:305] Clearing existing system tablet
I20260929 15:45:07.713083 7029 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:07.713193 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:07.717166 7029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 15:45:07.719488 7134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:07.720234 7029 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:07.720559 7029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 15:45:07.720660 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
uuid: "115035d166124ced826936fab2b13803"
format_stamp: "Formatted at 2026-09-29 15:45:07 on dist-test-slave-7491"
server_key: "f61ea075984120dd536812cf0596569a"
server_key_iv: "6937184c4604fbff113be453e9925434"
server_key_version: "encryptionkey@0"
I20260929 15:45:07.726476 7029 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:07.736481 7029 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:45:07.736719 7029 log.cc:1196] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 15:45:07.737210 7029 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Deleting consensus metadata
I20260929 15:45:07.738049 7029 master_runner.cc:315] Copying system tablet from 127.0.97.62:38733
I20260929 15:45:07.758461 7029 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:38733
I20260929 15:45:07.790822 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.759861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.0.97.62:38733)
0929 15:45:07.760301 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:07.760309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:07.764920 (+ 4611us) server_negotiation.cc:407] Connection header received
0929 15:45:07.765268 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:07.765274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:07.765397 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:07.765477 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:07.767278 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.768508 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.770563 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.771472 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.771735 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.782441 (+ 10706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:07.782449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:07.782454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:07.782502 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:07.787060 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.788213 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.788234 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.788237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.788443 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:07.788965 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:07.788971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:07.788975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:07.789401 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:07.789539 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:07.790144 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 15:45:07.790426 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260929 15:45:07.792352 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:07.759692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52352)
0929 15:45:07.764733 (+ 5041us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:07.764762 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:07.765071 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:07.765723 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:07.765728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:07.766323 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:07.767061 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.767078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.768739 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.768744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.770342 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:07.770362 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:07.772068 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:07.772074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:07.772361 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:07.780257 (+ 7896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:07.780301 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:07.782002 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:07.787332 (+ 5330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.787340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.787345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.787975 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.788616 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:07.788620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:07.788625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:07.788809 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:07.789636 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:07.789642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:07.789835 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 15:45:07.791122 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0929 15:45:07.791547 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4837,"thread_start_us":4709,"threads_started":1}
I20260929 15:45:07.793743 6586 tablet_copy_service.cc:140] P bdfc10a7ac2c4cfebb70413ae3961488: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 115035d166124ced826936fab2b13803 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52352)
I20260929 15:45:07.794080 6586 tablet_copy_service.cc:161] P bdfc10a7ac2c4cfebb70413ae3961488: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 115035d166124ced826936fab2b13803 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52352: session id = 115035d166124ced826936fab2b13803-00000000000000000000000000000000
I20260929 15:45:07.800285 6586 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:45:07.811156 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:45:07.821455 7029 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: tablet copy: Starting download of 0 data blocks...
I20260929 15:45:07.821931 7029 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: tablet copy: Starting download of 1 WAL segments...
I20260929 15:45:07.826601 7029 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:45:07.833047 6585 tablet_copy_service.cc:342] P bdfc10a7ac2c4cfebb70413ae3961488: Request end of tablet copy session 115035d166124ced826936fab2b13803-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52352
I20260929 15:45:07.833348 6585 tablet_copy_service.cc:433] P bdfc10a7ac2c4cfebb70413ae3961488: ending tablet copy session 115035d166124ced826936fab2b13803-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 115035d166124ced826936fab2b13803
I20260929 15:45:07.848399 7029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:07.857061 7141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:07.869153 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:07.874428 7029 server_base.cc:1034] running on GCE node
W20260929 15:45:07.875444 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:07.876662 7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:07.877812 7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:07.879122 7029 hybrid_clock.cc:648] HybridClock initialized: now 1790696707879058 us; error 72 us; skew 500 ppm
I20260929 15:45:07.879523 7029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:07.881356 7023 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
Sep 29 15:45:07 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:07.882899 7029 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 15:45:07.889398 7029 webserver.cc:460] Webserver started at http://127.0.97.61:43559/ using document root <none> and password file <none>
I20260929 15:45:07.889953 7029 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:07.890074 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:07.894518 7029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:45:07.897284 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:07.898597 7029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:45:07.898818 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
uuid: "115035d166124ced826936fab2b13803"
format_stamp: "Formatted at 2026-09-29 15:45:07 on dist-test-slave-7491"
server_key: "f61ea075984120dd536812cf0596569a"
server_key_iv: "6937184c4604fbff113be453e9925434"
server_key_version: "encryptionkey@0"
I20260929 15:45:07.899081 7029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:07.927770 7029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:07.933909 7029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:07.962855 7029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38801
I20260929 15:45:07.962857 7202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38801 every 8 connection(s)
I20260929 15:45:07.964393 7029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:45:07.970816 7203 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:45:07.975587 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Bootstrap starting.
I20260929 15:45:08.000681 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:45:08.001847 7203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Bootstrap complete.
I20260929 15:45:08.003279 7203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } }
I20260929 15:45:08.003611 7203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 115035d166124ced826936fab2b13803, State: Initialized, Role: LEARNER
I20260929 15:45:08.003898 7203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } }
I20260929 15:45:08.004835 7206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } }
I20260929 15:45:08.005331 7203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:08.005354 7206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:45:08.016409 7217 catalog_manager.cc:1259] Loaded cluster ID: 1e8830dd33ae4aafac80e4f49874a95c
I20260929 15:45:08.016563 7217 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: loading cluster ID for follower catalog manager: success
I20260929 15:45:08.019469 7217 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: acquiring CA information for follower catalog manager: success
I20260929 15:45:08.021831 7217 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:45:08.206952 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:08.207304 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:08.207408 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:08.217602 7125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:08.218235 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:08.218374 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:08.218493 7125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:08.218581 7125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:08.230465 7125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:08.232033 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:08.233510 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:08.250561 7225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:08.250535 7223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:08.250506 7222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:08.251278 7125 server_base.cc:1034] running on GCE node
I20260929 15:45:08.252492 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:08.254643 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:08.255961 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790696708255860 us; error 103 us; skew 500 ppm
I20260929 15:45:08.256631 7125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:08.261114 7125 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 15:45:08.264127 7125 webserver.cc:460] Webserver started at http://127.0.97.60:34699/ using document root <none> and password file <none>
I20260929 15:45:08.265606 7125 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:08.265791 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:08.266355 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:08.270952 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/instance:
uuid: "39f9b121986f4071b281bdab26c1c316"
format_stamp: "Formatted at 2026-09-29 15:45:08 on dist-test-slave-7491"
server_key: "82baaf42de93995690c2240ca0cd9f65"
server_key_iv: "16ee61b2d0ad3249230b5cb30961b939"
server_key_version: "encryptionkey@0"
I20260929 15:45:08.272372 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal/instance:
uuid: "39f9b121986f4071b281bdab26c1c316"
format_stamp: "Formatted at 2026-09-29 15:45:08 on dist-test-slave-7491"
server_key: "82baaf42de93995690c2240ca0cd9f65"
server_key_iv: "16ee61b2d0ad3249230b5cb30961b939"
server_key_version: "encryptionkey@0"
I20260929 15:45:08.273984 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.263846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39622)
0929 15:45:08.265146 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.265173 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.265329 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.266407 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.266413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.266439 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:08.266642 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.266651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.268718 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.268727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.271019 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.271048 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.273018 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.273026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.273234 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.273312 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.273453 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.273634 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":1057,"threads_started":1}
I20260929 15:45:08.274005 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.263941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.0.97.61:38801)
0929 15:45:08.265873 (+ 1932us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.265879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.265904 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:45:08.265981 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.265986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.266088 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.266256 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:08.266797 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.268468 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.271269 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.272578 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.272977 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.273286 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.273521 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.273678 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":1648,"threads_started":1}
I20260929 15:45:08.275638 7178 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 15:45:08.276434 7206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdfc10a7ac2c4cfebb70413ae3961488. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } }
I20260929 15:45:08.276814 7206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:45:08.281356 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20260929 15:45:08.286655 7235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:08.288940 7125 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:45:08.289237 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
uuid: "39f9b121986f4071b281bdab26c1c316"
format_stamp: "Formatted at 2026-09-29 15:45:08 on dist-test-slave-7491"
server_key: "82baaf42de93995690c2240ca0cd9f65"
server_key_iv: "16ee61b2d0ad3249230b5cb30961b939"
server_key_version: "encryptionkey@0"
I20260929 15:45:08.289520 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:08.316560 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:08.323217 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:08.323953 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:08.350831 7125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40467
I20260929 15:45:08.350970 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40467 every 8 connection(s)
I20260929 15:45:08.352716 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:45:08.356964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 7125
I20260929 15:45:08.358243 7287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:08.371287 7287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:08.389130 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.360098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.97.60:40467)
0929 15:45:08.361247 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.361263 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.361318 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:45:08.363370 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.363415 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.364287 (+ 872us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.364986 (+ 699us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.367073 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.369144 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.371107 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.372599 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.372839 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.379306 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.379314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.379344 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.379388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.383170 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.384413 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.384425 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.384439 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.384773 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.385352 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.385360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.385364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.386300 (+ 936us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.386661 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.387212 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.387691 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":504,"threads_started":1}
I20260929 15:45:08.393656 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.375657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.0.97.62:38733)
0929 15:45:08.377550 (+ 1893us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.377554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.377571 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:45:08.377625 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.377628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.377701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.377838 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.379251 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.380108 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.381785 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.382249 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.382420 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.387008 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.387015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.387020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.387071 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.390430 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.391357 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.391365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.391371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.391522 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.391924 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.391931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.391939 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.392330 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.392513 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.392979 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.393226 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1656,"threads_started":1}
I20260929 15:45:08.393680 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.375530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52362)
0929 15:45:08.376796 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.376865 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.377086 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.377933 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.377943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.378756 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.379025 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.379035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.380309 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.380317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.381591 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.381607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.382479 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.382485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.382691 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.383700 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.383730 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.386597 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.390615 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.390628 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.390636 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.391183 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.391651 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.391656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.391660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.391796 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.392485 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.392493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.392668 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.392867 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.393227 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":549,"threads_started":1}
I20260929 15:45:08.395366 7287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 15:45:08.411338 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.396370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39624)
0929 15:45:08.396825 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.396849 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.397105 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.397487 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.397492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.397910 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.398065 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.398073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.399158 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.399162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.400102 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.400113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.400944 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.400950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.401159 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.401871 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.401889 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.404585 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.408558 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.408564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.408569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.409053 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.409501 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.409507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.409511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.409666 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.410458 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.410462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.410618 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.410753 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.411082 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260929 15:45:08.411535 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.396590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.0.97.61:38801)
0929 15:45:08.397023 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.397032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.397065 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:45:08.397279 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.397284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.397356 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.397461 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.398222 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.398961 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.400314 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.400757 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.400910 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.404853 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.404859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.404862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.404906 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.408373 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.409203 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.409208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.409210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.409377 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.409921 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.409926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.409928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.410295 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.410463 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.410980 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.411202 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260929 15:45:08.412657 7287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:45:08.427790 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.413444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.0.1:37334)
0929 15:45:08.413811 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.413837 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.414014 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.414392 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.414396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.414908 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.415112 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.415121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.416345 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.416350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.417515 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.417532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.418396 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.418402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.418591 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.419391 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.419413 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.421863 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.425254 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.425259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.425262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.425682 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.426184 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.426188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.426191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.426353 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.426939 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.426943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.427099 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.427239 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.427481 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 15:45:08.427958 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.413660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.0.97.60:40467)
0929 15:45:08.413833 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.413837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.413944 (+ 107us) server_negotiation.cc:407] Connection header received
0929 15:45:08.414131 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.414135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.414198 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.414301 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.415350 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.416078 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.417718 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.418152 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.418270 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.422136 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.422142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.422145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.422178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.425047 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.425894 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.425899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.425901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.426045 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.426467 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.426471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.426473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.426787 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.426919 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.427359 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.427629 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":54}
I20260929 15:45:08.435151 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Bootstrap starting.
I20260929 15:45:08.440569 7287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:08.442291 7287 log.cc:824] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:08.445974 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: No bootstrap required, opened a new log
I20260929 15:45:08.453796 7287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } }
I20260929 15:45:08.454170 7287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:08.454262 7287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f9b121986f4071b281bdab26c1c316, State: Initialized, Role: FOLLOWER
I20260929 15:45:08.454851 7287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } }
I20260929 15:45:08.455866 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } } }
I20260929 15:45:08.456310 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:45:08.457103 7287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:45:08.469039 7305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:45:08.469334 7305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:45:08.477697 7125 master_runner.cc:428] Detected that this master 39f9b121986f4071b281bdab26c1c316 is joining an existing cluster
I20260929 15:45:08.477876 7125 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:40467
I20260929 15:45:08.498188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6627
I20260929 15:45:08.511562 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.483699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.0.97.61:38801)
0929 15:45:08.483917 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.483925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.485167 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 15:45:08.485437 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.485446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.485537 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.485617 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.487291 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.488819 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.491310 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.492266 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.492456 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.495870 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.495878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.495883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.495929 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.501441 (+ 5512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.508606 (+ 7165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.508616 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.508631 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.508837 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.509500 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.509504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.509509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.509908 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.510038 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.510901 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.511184 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:45:08.511960 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.483554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.97.62:38733)
0929 15:45:08.483802 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.483809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.485869 (+ 2060us) server_negotiation.cc:407] Connection header received
0929 15:45:08.486178 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.486185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.486274 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.486353 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.488333 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.489882 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.492967 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.494839 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.495023 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.501238 (+ 6215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.501248 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.501254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.501303 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.506189 (+ 4886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.507833 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.507842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.507848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.508903 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.509467 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.509475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.509479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.509898 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.510038 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.511327 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.511597 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 15:45:08.512995 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.483623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39636)
0929 15:45:08.484980 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.485013 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.485218 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.485862 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.485867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.486661 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.487057 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.487071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.489286 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.489292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.491090 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.491106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.492970 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.492977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.493180 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.494223 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.494250 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.495577 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.507363 (+ 11786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.507368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.507372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.507935 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.509071 (+ 1136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.509075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.509078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.509289 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.510201 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.510208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.510479 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.511873 (+ 1394us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.512385 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":4,"spinlock_wait_cycles":60928,"thread_start_us":548,"threads_started":1}
I20260929 15:45:08.513036 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.484322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52378)
0929 15:45:08.485752 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.485773 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.485986 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.486599 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.486606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.487540 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.488041 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.488060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.490144 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.490152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.492691 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.492716 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.498269 (+ 5553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.498277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.498907 (+ 630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.499679 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.499708 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.500934 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.506476 (+ 5542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.506485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.506492 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.507656 (+ 1164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.509047 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.509056 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.509059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.509278 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.510127 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.510134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.510479 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.510639 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.511035 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":32,"spinlock_wait_cycles":44288,"thread_start_us":1037,"threads_started":1}
I20260929 15:45:08.521561 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.485870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.0.1:37346)
0929 15:45:08.497469 (+ 11599us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.497496 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.497672 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.499409 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.499415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.500761 (+ 1346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.501216 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.501231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.502746 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.502751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.504429 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.504456 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.506089 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.506098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.506341 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.509244 (+ 2903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.509279 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.510772 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.516850 (+ 6078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.516856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.516861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.517469 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.518339 (+ 870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.518347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.518352 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.518543 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.519589 (+ 1046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.519594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.520253 (+ 659us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.520483 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.520906 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11377,"thread_start_us":11930,"threads_started":1}
I20260929 15:45:08.522699 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.486213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.0.97.60:40467)
0929 15:45:08.486422 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.486427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.497617 (+ 11190us) server_negotiation.cc:407] Connection header received
0929 15:45:08.498608 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.498614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.498728 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.498861 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.501381 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.502539 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.504615 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.505323 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.505458 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.510992 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.511000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.511008 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.511044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.516681 (+ 5637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.517627 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.517631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.517633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.518089 (+ 456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.518726 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.518734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.518739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.519272 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.519524 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.521987 (+ 2463us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.522251 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:45:08.540802 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.500415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.97.1:45041)
0929 15:45:08.507423 (+ 7008us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.507434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.507470 (+ 36us) server_negotiation.cc:407] Connection header received
0929 15:45:08.507552 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.507562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.508462 (+ 900us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.510744 (+ 2282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.511036 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.512873 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.516317 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.517260 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.519116 (+ 1856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.531180 (+ 12064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.531199 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.531216 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.531255 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.536905 (+ 5650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.537961 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.537973 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.537977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.538179 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.538693 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.538699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.538702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.539249 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.539567 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.539986 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.540327 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6802,"thread_start_us":6611,"threads_started":1}
I20260929 15:45:08.541260 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.521862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.0.1:37356)
0929 15:45:08.522749 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.522775 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.522925 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.523467 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.523474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.524018 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.524317 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.524333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.526119 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.526126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.527589 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.527606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.529139 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.529147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.529384 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.530240 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.530265 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.534264 (+ 3999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.538325 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.538333 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.538340 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.538889 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.539300 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.539305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.539309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.539488 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.540253 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.540260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.540453 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.540611 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.540869 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724}
I20260929 15:45:08.542840 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.522236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.0.97.60:40467)
0929 15:45:08.522945 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.522951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.522971 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:45:08.523095 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.523102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.523229 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.523345 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.524578 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.525894 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.527836 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.528896 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.529764 (+ 868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.534674 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.534680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.534684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.534725 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.538153 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.539030 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.539035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.539040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.539170 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.539632 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.539637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.539640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.540018 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.541616 (+ 1598us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.541737 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.541926 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":8584,"threads_started":1}
I20260929 15:45:08.542865 6750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:08.544865 6565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:40467: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "39f9b121986f4071b281bdab26c1c316"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 40467
}
attrs {
promote: true
}
}
dest_uuid: "bdfc10a7ac2c4cfebb70413ae3961488"
cas_config_opid_index: 5
I20260929 15:45:08.546494 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } }
I20260929 15:45:08.549157 7228 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 39f9b121986f4071b281bdab26c1c316 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.557706 7262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:08.563364 7228 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:40467
I20260929 15:45:08.564265 7125 master.cc:537] Master@127.0.97.60:40467 shutting down...
I20260929 15:45:08.564265 7262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Refusing update from remote peer bdfc10a7ac2c4cfebb70413ae3961488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:45:08.566931 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 LEARNER]: Refusing update from remote peer bdfc10a7ac2c4cfebb70413ae3961488: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:45:08.576594 7317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:45:08.577530 7319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 15:45:08.578838 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.579411 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
W20260929 15:45:08.580060 7241 tablet.cc:2369] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:08.585454 7178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 39f9b121986f4071b281bdab26c1c316 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.590785 7206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: true } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.591123 7206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:45:08.592538 7318 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: attempting to promote NON_VOTER 115035d166124ced826936fab2b13803 to VOTER
I20260929 15:45:08.593465 7318 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } }
I20260929 15:45:08.596421 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 LEARNER]: Refusing update from remote peer bdfc10a7ac2c4cfebb70413ae3961488: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 15:45:08.597188 7319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 15:45:08.601032 7317 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 115035d166124ced826936fab2b13803 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.602766 7178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 115035d166124ced826936fab2b13803 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.607767 7125 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:08.608266 7125 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:08.608479 7125 tablet_replica.cc:330] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: stopping tablet replica
I20260929 15:45:08.609082 7206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.609390 7206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:45:08.618829 7125 master.cc:552] Master@127.0.97.60:40467 shutdown complete.
I20260929 15:45:08.618976 7125 master_runner.cc:305] Clearing existing system tablet
I20260929 15:45:08.619565 7125 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:08.619729 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:08.623116 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.623481 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:08.624084 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.624568 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:08.627305 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.626515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.0.1:37370)
0929 15:45:08.626826 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.627012 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 15:45:08.627650 6550 consensus_peers.cc:487] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 -> Peer 39f9b121986f4071b281bdab26c1c316 (127.0.97.60:40467): Couldn't send request to peer 39f9b121986f4071b281bdab26c1c316. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40467: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:45:08.629535 7125 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.008s
I20260929 15:45:08.631250 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:08.631922 7125 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:08.632385 7125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 15:45:08.632513 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
uuid: "39f9b121986f4071b281bdab26c1c316"
format_stamp: "Formatted at 2026-09-29 15:45:08 on dist-test-slave-7491"
server_key: "82baaf42de93995690c2240ca0cd9f65"
server_key_iv: "16ee61b2d0ad3249230b5cb30961b939"
server_key_version: "encryptionkey@0"
I20260929 15:45:08.637440 7125 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:08.644871 7125 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:45:08.645087 7125 log.cc:1196] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 15:45:08.645660 7125 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Deleting consensus metadata
I20260929 15:45:08.645970 7125 master_runner.cc:315] Copying system tablet from 127.0.97.62:38733
I20260929 15:45:08.648913 7125 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:38733
I20260929 15:45:08.666126 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.649787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.0.97.62:38733)
0929 15:45:08.650012 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:08.650022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:08.650522 (+ 500us) server_negotiation.cc:407] Connection header received
0929 15:45:08.650790 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:08.650796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:08.650891 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:08.651007 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:08.652235 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.653371 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.655146 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.655647 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.655799 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.659001 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:08.659007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:08.659011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:08.659048 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:08.663040 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.663833 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.663840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.663844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.663993 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:08.664505 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:08.664514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:08.664519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:08.664859 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:08.665025 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:08.665476 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 15:45:08.665720 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:45:08.666201 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:08.649784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52390)
0929 15:45:08.650407 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:08.650424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:08.650605 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:08.651034 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:08.651038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:08.651523 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:08.651741 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.651754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.653830 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.653836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.654961 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:08.654977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:08.655827 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:08.655833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:08.656317 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:08.657174 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:08.657196 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:08.658667 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:08.663245 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.663251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.663256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.663687 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.664118 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:08.664123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:08.664127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:08.664381 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:08.665001 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:08.665006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:08.665198 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 15:45:08.665365 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 15:45:08.665742 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":361,"threads_started":1}
I20260929 15:45:08.667214 6585 tablet_copy_service.cc:140] P bdfc10a7ac2c4cfebb70413ae3961488: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 39f9b121986f4071b281bdab26c1c316 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52390)
I20260929 15:45:08.667397 6585 tablet_copy_service.cc:161] P bdfc10a7ac2c4cfebb70413ae3961488: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 39f9b121986f4071b281bdab26c1c316 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52390: session id = 39f9b121986f4071b281bdab26c1c316-00000000000000000000000000000000
I20260929 15:45:08.670095 6585 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:45:08.672775 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:45:08.678532 7125 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: tablet copy: Starting download of 0 data blocks...
I20260929 15:45:08.678886 7125 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: tablet copy: Starting download of 1 WAL segments...
I20260929 15:45:08.682217 7125 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:45:08.686967 6585 tablet_copy_service.cc:342] P bdfc10a7ac2c4cfebb70413ae3961488: Request end of tablet copy session 39f9b121986f4071b281bdab26c1c316-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52390
I20260929 15:45:08.687214 6585 tablet_copy_service.cc:433] P bdfc10a7ac2c4cfebb70413ae3961488: ending tablet copy session 39f9b121986f4071b281bdab26c1c316-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 39f9b121986f4071b281bdab26c1c316
I20260929 15:45:08.700239 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:08.703760 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:08.712777 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6627
W20260929 15:45:08.714049 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:08.715588 7125 server_base.cc:1034] running on GCE node
W20260929 15:45:08.716645 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:08.717278 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:08.718025 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:08.719442 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790696708719370 us; error 37 us; skew 500 ppm
I20260929 15:45:08.719827 7125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:08 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:08.722995 7125 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 15:45:08.729341 7125 webserver.cc:460] Webserver started at http://127.0.97.60:37951/ using document root <none> and password file <none>
I20260929 15:45:08.729729 7125 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:08.730120 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:08.737419 7125 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.006s
I20260929 15:45:08.739957 7342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:08.740217 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45041
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38733
--webserver_port=34461
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:08.741072 7125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 15:45:08.741314 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
uuid: "39f9b121986f4071b281bdab26c1c316"
format_stamp: "Formatted at 2026-09-29 15:45:08 on dist-test-slave-7491"
server_key: "82baaf42de93995690c2240ca0cd9f65"
server_key_iv: "16ee61b2d0ad3249230b5cb30961b939"
server_key_version: "encryptionkey@0"
I20260929 15:45:08.741536 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:08.763865 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:08.768997 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:08.801425 7125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40467
I20260929 15:45:08.801465 7394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40467 every 8 connection(s)
I20260929 15:45:08.802608 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:45:08.808905 7395 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:45:08.813588 7395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Bootstrap starting.
I20260929 15:45:08.846689 7395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:45:08.847266 7395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: Bootstrap complete.
I20260929 15:45:08.848419 7395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } }
I20260929 15:45:08.848733 7395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 39f9b121986f4071b281bdab26c1c316, State: Initialized, Role: LEARNER
I20260929 15:45:08.849007 7395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } }
I20260929 15:45:08.850371 7395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:08.850265 7398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:08.850790 7398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:45:08.861158 7409 catalog_manager.cc:1259] Loaded cluster ID: 1e8830dd33ae4aafac80e4f49874a95c
I20260929 15:45:08.861382 7409 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: loading cluster ID for follower catalog manager: success
I20260929 15:45:08.865545 7409 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: acquiring CA information for follower catalog manager: success
I20260929 15:45:08.868949 7409 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:45:09.030995 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.021299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.0.1:37386)
0929 15:45:09.021752 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:09.021775 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:09.021909 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:09.023472 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:09.023480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:09.023524 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:09.023816 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.023832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.026044 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.026053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.028028 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.028052 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.030093 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.030101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.030339 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.030380 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:45:09.030495 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 15:45:09.030664 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298}
I20260929 15:45:09.031062 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.022141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.97.60:40467)
0929 15:45:09.023030 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:09.023036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:09.023059 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:45:09.023137 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:09.023142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:09.023242 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:09.023382 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:09.024036 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.025837 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.028313 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.029837 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.029999 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.030283 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:09.030541 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 15:45:09.030692 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":560,"threads_started":1}
I20260929 15:45:09.032452 7370 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 15:45:09.033329 7398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdfc10a7ac2c4cfebb70413ae3961488. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:09.033639 7398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: This master's current role is: LEARNER
W20260929 15:45:09.200485 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:09.200960 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:09.201153 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:09.209695 7345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:09.209890 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:09.210067 7345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:09.221630 7345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34461
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:09.223194 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:09.224717 7345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:09.242336 7416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:09.242465 7415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:09.242374 7418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:09.243165 7345 server_base.cc:1034] running on GCE node
I20260929 15:45:09.243932 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:09.245306 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:09.246835 7345 hybrid_clock.cc:648] HybridClock initialized: now 1790696709246739 us; error 95 us; skew 500 ppm
I20260929 15:45:09.247382 7345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:09 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:09.251448 7345 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:45:09.254047 7345 webserver.cc:460] Webserver started at http://127.0.97.1:34461/ using document root <none> and password file <none>
I20260929 15:45:09.254871 7345 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:09.255146 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:09.263595 7345 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:45:09.267392 7425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:09.269277 7345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260929 15:45:09.269481 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "77332a715cbc4e62ac3edf365ad3b939"
format_stamp: "Formatted at 2026-09-29 15:45:05 on dist-test-slave-7491"
server_key: "7806db3083791f855b0db65059e7e0eb"
server_key_iv: "9ee4e58a8fe9c0cb54d8f7076614830c"
server_key_version: "encryptionkey@0"
I20260929 15:45:09.270560 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:09.305655 7345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:09.311079 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:09.311421 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:09.312821 7345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:09.315094 7345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:09.315218 7345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:09.315343 7345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:09.315430 7345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:09.364929 7345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45041
I20260929 15:45:09.365204 7537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45041 every 8 connection(s)
I20260929 15:45:09.366658 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:09.376961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 7345
I20260929 15:45:09.377312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6761
Sep 29 15:45:09 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 15:45:09 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 29 15:45:09 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 15:45:09 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:09.416759 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.371619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41799 (local address 127.0.97.60:40467)
0929 15:45:09.371804 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:09.371810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:09.382478 (+ 10668us) server_negotiation.cc:407] Connection header received
0929 15:45:09.390623 (+ 8145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:09.390634 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:09.390760 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:09.390883 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:09.394382 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.395766 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.398539 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.399306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.399438 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.405362 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:09.405371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:09.405375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:09.405423 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:09.409831 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.411282 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.411291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.411296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.411503 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.412233 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.412241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.412245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.412659 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:09.412826 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:09.416015 (+ 3189us) server_negotiation.cc:299] Negotiation successful
0929 15:45:09.416365 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:45:09.416860 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.372120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.97.1:41799)
0929 15:45:09.382156 (+ 10036us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:09.382263 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:09.389005 (+ 6742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:09.391591 (+ 2586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:09.391613 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:09.392845 (+ 1232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:09.394041 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.394076 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.396114 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.396122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.398334 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.398349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.400096 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.400105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.400494 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.401521 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:09.401553 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:09.404978 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:09.410094 (+ 5116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:09.410111 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:09.410143 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:09.411074 (+ 931us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:09.411667 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:09.411679 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:09.411687 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:09.412034 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:09.412921 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:09.412932 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:09.413618 (+ 686us) client_negotiation.cc:769] Sending connection context
0929 15:45:09.414015 (+ 397us) client_negotiation.cc:240] Negotiation successful
0929 15:45:09.414933 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9477,"thread_start_us":17489,"threads_started":1}
I20260929 15:45:09.418956 7538 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40467
I20260929 15:45:09.419430 7538 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:09.420565 7538 heartbeater.cc:511] Master 127.0.97.60:40467 requested a full tablet report, sending...
I20260929 15:45:09.423682 7360 ts_manager.cc:194] Registered new tserver with Master: 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041)
I20260929 15:45:09.425228 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.372120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.97.1:46085)
0929 15:45:09.390124 (+ 18004us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:09.390146 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:09.390285 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:09.402131 (+ 11846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:09.402151 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:09.402985 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:09.403323 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.403336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.405065 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.405070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.406661 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.406678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.408905 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.408933 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.409100 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.409876 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:09.409903 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:09.413464 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:09.421409 (+ 7945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:09.421429 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:09.421433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:09.422347 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:09.422905 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:09.422909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:09.422913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:09.423074 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:09.423737 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:09.423742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:09.424057 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 15:45:09.424243 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 15:45:09.424721 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17794,"mutex_wait_us":53,"spinlock_wait_cycles":27904,"thread_start_us":10198,"threads_started":1}
I20260929 15:45:09.425509 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.379240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.0.97.2:45411)
0929 15:45:09.395633 (+ 16393us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:09.395648 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:09.397564 (+ 1916us) server_negotiation.cc:407] Connection header received
0929 15:45:09.397674 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:09.397686 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:09.398059 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:09.398314 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:09.400736 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.402425 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.405429 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.406516 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.406668 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.413935 (+ 7267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:09.413960 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:09.413984 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:09.414038 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:09.419725 (+ 5687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.421432 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.421446 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.421449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.421683 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.422528 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.422535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.422539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.423081 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:09.423424 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:09.424737 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0929 15:45:09.425038 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16048,"thread_start_us":16312,"threads_started":1}
I20260929 15:45:09.425691 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.390381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:44159)
0929 15:45:09.397104 (+ 6723us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:09.397129 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:09.397265 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:09.403503 (+ 6238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:09.403510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:09.404088 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:09.404492 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.404504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.406888 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.406893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.408790 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:09.408813 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.411833 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.411843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:09.412065 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.413933 (+ 1868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:09.413960 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:09.418107 (+ 4147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:09.422803 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:09.422810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:09.422816 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:09.423377 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:09.423849 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:09.423854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:09.423857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:09.424035 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:09.424790 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:09.424795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:09.424969 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 15:45:09.425115 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:45:09.425374 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6583,"spinlock_wait_cycles":12032,"thread_start_us":6972,"threads_started":1}
I20260929 15:45:09.426484 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.390584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44159 (local address 127.0.97.62:38733)
0929 15:45:09.403144 (+ 12560us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:09.403150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:09.403171 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:45:09.403239 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:09.403246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:09.403331 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:09.403458 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:09.405302 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.406740 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.409595 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.411037 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.411212 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.418429 (+ 7217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:09.418437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:09.418441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:09.418493 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:09.422515 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.423542 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.423548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.423551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.423701 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.424224 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.424229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.424231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.424590 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:09.424732 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:09.425888 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0929 15:45:09.426127 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12421,"thread_start_us":12974,"threads_started":1}
I20260929 15:45:09.427037 6884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:09.427304 7539 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260929 15:45:09.427469 7539 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:09.427532 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:09.371059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46085 (local address 127.0.97.61:38801)
0929 15:45:09.401487 (+ 30428us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:09.401497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:09.401534 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:45:09.401631 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:09.401637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:09.401760 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:09.403523 (+ 1763us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:09.403696 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.404854 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.407602 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:09.408327 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:09.411483 (+ 3156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:09.413797 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:09.413805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:09.413810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:09.413872 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:09.421191 (+ 7319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.422568 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.422574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.422577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.422758 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:09.423229 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:09.423234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:09.423237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:09.423575 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:09.426284 (+ 2709us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:09.426409 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 15:45:09.426974 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30240,"thread_start_us":30832,"threads_started":1}
I20260929 15:45:09.428076 7539 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260929 15:45:09.428562 7540 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38801
I20260929 15:45:09.428735 7540 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:09.429417 6565 ts_manager.cc:194] Re-registered known tserver with Master: 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041)
I20260929 15:45:09.429739 7540 heartbeater.cc:511] Master 127.0.97.61:38801 requested a full tablet report, sending...
I20260929 15:45:09.430459 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44159
I20260929 15:45:09.432080 7168 ts_manager.cc:194] Registered new tserver with Master: 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041)
I20260929 15:45:09.526340 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6761
I20260929 15:45:09.546957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45411
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38733
--webserver_port=39415
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:09.620855 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: true } } }
I20260929 15:45:09.621281 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:09.621892 7552 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: attempting to promote NON_VOTER 39f9b121986f4071b281bdab26c1c316 to VOTER
I20260929 15:45:09.622841 7552 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:09.625766 7370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 LEARNER]: Refusing update from remote peer bdfc10a7ac2c4cfebb70413ae3961488: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:45:09.625779 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Refusing update from remote peer bdfc10a7ac2c4cfebb70413ae3961488: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:45:09.626478 7552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:45:09.626948 7551 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:45:09.630419 7552 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 39f9b121986f4071b281bdab26c1c316 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.631309 7178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 39f9b121986f4071b281bdab26c1c316 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.635680 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.636034 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:09.636595 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.636929 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:09.637249 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.637459 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:09.637010 7370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 39f9b121986f4071b281bdab26c1c316 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.640872 7557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.641945 7557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:45:09.643841 7558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:09.644223 7558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 15:45:09.984359 7550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:09.984949 7550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:09.985100 7550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:09.994091 7550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:09.994263 7550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:09.994395 7550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:10.007442 7550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39415
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:10.009078 7550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:10.010846 7550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:10.027527 7563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:10.027640 7564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:10.027597 7566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:10.029706 7550 server_base.cc:1034] running on GCE node
I20260929 15:45:10.030493 7550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:10.031631 7550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:10.032992 7550 hybrid_clock.cc:648] HybridClock initialized: now 1790696710032922 us; error 67 us; skew 500 ppm
I20260929 15:45:10.033422 7550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:10.038170 7550 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:45:10.040934 7550 webserver.cc:460] Webserver started at http://127.0.97.2:39415/ using document root <none> and password file <none>
I20260929 15:45:10.042012 7550 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:10.042241 7550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:10.049238 7550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:10.053453 7573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:10.055843 7550 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:45:10.056123 7550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "09a0b7a360cb482e8388398c020630c8"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "ed1b0dedb3a4ef860491369f1d2a33ca"
server_key_iv: "4ebb3b63423a4e5bcac9cfb65c14014d"
server_key_version: "encryptionkey@0"
I20260929 15:45:10.057574 7550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:10.098997 7550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:10.105509 7550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:10.105927 7550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:10.107098 7550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:10.108783 7550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:10.108919 7550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:10.109067 7550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:10.109157 7550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:10.156251 7550 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45411
I20260929 15:45:10.156364 7686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45411 every 8 connection(s)
I20260929 15:45:10.158212 7550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:10.162864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 7550
I20260929 15:45:10.163189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6895
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:10.197798 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.165635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36737 (local address 127.0.97.60:40467)
0929 15:45:10.172634 (+ 6999us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.172639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.172694 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:45:10.172763 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.172769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.172846 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.172976 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.175262 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.176625 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.179656 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.180390 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.180525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.186954 (+ 6429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.186962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.186967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.187014 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.191815 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.193014 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.193023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.193028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.193194 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.193756 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.193765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.193770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.194231 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.194524 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.197013 (+ 2489us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.197336 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6868,"thread_start_us":6753,"threads_started":1}
I20260929 15:45:10.201524 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.168690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.97.2:36737)
0929 15:45:10.170068 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:10.170095 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:10.170242 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:10.173334 (+ 3092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:10.173346 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:10.174152 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:10.174961 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.174998 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.176847 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.176853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.179416 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.179441 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.180862 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.180868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.181190 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.182277 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:10.182314 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:10.186529 (+ 4215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:10.192015 (+ 5486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.192029 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.192054 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.192842 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.193337 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.193342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.193346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.193620 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.194411 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:10.194421 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:10.196429 (+ 2008us) client_negotiation.cc:769] Sending connection context
0929 15:45:10.198557 (+ 2128us) client_negotiation.cc:240] Negotiation successful
0929 15:45:10.199528 (+ 971us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"thread_start_us":1094,"threads_started":1}
I20260929 15:45:10.202966 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.163295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.97.2:56255)
0929 15:45:10.169334 (+ 6039us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:10.169390 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:10.169563 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:10.176837 (+ 7274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:10.176845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:10.177786 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:10.178786 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.178808 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.181078 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.181087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.183483 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.183510 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.184919 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.184927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.185165 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.186078 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:10.186101 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:10.190844 (+ 4743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:10.196868 (+ 6024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.196876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.196881 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.197579 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.199279 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.199286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.199291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.199494 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.200459 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:10.200464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:10.200826 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 15:45:10.201010 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 15:45:10.202433 (+ 1423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5839,"mutex_wait_us":453,"thread_start_us":5180,"threads_started":1}
I20260929 15:45:10.203259 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.164941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.97.3:34081)
0929 15:45:10.175800 (+ 10859us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.175809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.176747 (+ 938us) server_negotiation.cc:407] Connection header received
0929 15:45:10.176826 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.176835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.177228 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.177395 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.179909 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.181454 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.183402 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.184418 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.184550 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.192645 (+ 8095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.192671 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.192697 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.192775 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.198177 (+ 5402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.199685 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.199700 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.199705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.199929 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.200637 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.200642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.200646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.201179 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.201512 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.202241 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.202771 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10632,"thread_start_us":10324,"threads_started":1}
I20260929 15:45:10.204324 7687 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40467
I20260929 15:45:10.204738 7687 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:10.204823 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.163296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:44679)
0929 15:45:10.164624 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:10.164683 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:10.168263 (+ 3580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:10.185043 (+ 16780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:10.185049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:10.186059 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:10.186392 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.186409 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.188047 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.188055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.190148 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.190174 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.191736 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.191744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.191945 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.192689 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:10.192716 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:10.196041 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:10.201055 (+ 5014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.201064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.201071 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.201738 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.202375 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.202383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.202387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.202647 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.203407 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:10.203412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:10.203657 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 15:45:10.203791 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 15:45:10.204376 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":573,"threads_started":1}
I20260929 15:45:10.205508 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.163560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56255 (local address 127.0.97.61:38801)
0929 15:45:10.176261 (+ 12701us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.176268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.176301 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:45:10.176363 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.176370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.176485 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.179041 (+ 2556us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.179300 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.180836 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.183936 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.184671 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.185571 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.191141 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.191151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.191155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.191410 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.196593 (+ 5183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.197802 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.197812 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.197817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.199087 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.199728 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.199733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.199736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.200299 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.204222 (+ 3923us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.204517 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.205105 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12523,"thread_start_us":13053,"threads_started":1}
I20260929 15:45:10.205814 7687 heartbeater.cc:511] Master 127.0.97.60:40467 requested a full tablet report, sending...
I20260929 15:45:10.205996 7018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:10.206624 7689 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38801
I20260929 15:45:10.206786 7689 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:10.207309 7689 heartbeater.cc:511] Master 127.0.97.61:38801 requested a full tablet report, sending...
I20260929 15:45:10.208820 7168 ts_manager.cc:194] Registered new tserver with Master: 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411)
I20260929 15:45:10.209686 7360 ts_manager.cc:194] Registered new tserver with Master: 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411)
I20260929 15:45:10.212787 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.162293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44679 (local address 127.0.97.62:38733)
0929 15:45:10.184648 (+ 22355us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.184657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.184678 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:45:10.184760 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.184765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.184842 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.185536 (+ 694us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.186599 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.187851 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.190758 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.191473 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.192313 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.196351 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.196359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.196364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.196407 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.200834 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.201978 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.201985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.201988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.202193 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.202821 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.202829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.202835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.203267 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.205226 (+ 1959us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.205480 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.212487 (+ 7007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22203,"thread_start_us":22440,"threads_started":1}
I20260929 15:45:10.213773 7688 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260929 15:45:10.214056 7688 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:10.214509 7688 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260929 15:45:10.216012 6565 ts_manager.cc:194] Re-registered known tserver with Master: 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411)
I20260929 15:45:10.217308 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44679
I20260929 15:45:10.302420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6895
I20260929 15:45:10.322731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34081
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38733
--webserver_port=39211
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:10.434669 7539 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
W20260929 15:45:10.755808 7700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:10.756274 7700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:10.756445 7700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:10.766870 7700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:10.767033 7700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:10.767259 7700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:10.780699 7700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39211
--tserver_master_addrs=127.0.97.62:38733,127.0.97.61:38801,127.0.97.60:40467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:10.782334 7700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:10.783843 7700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:10.800356 7706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:10.801839 7700 server_base.cc:1034] running on GCE node
W20260929 15:45:10.804461 7705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:10.804824 7708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:10.805764 7700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:10.807180 7700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:10.808451 7700 hybrid_clock.cc:648] HybridClock initialized: now 1790696710808388 us; error 67 us; skew 500 ppm
I20260929 15:45:10.808863 7700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:10.820384 7700 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:45:10.823007 7700 webserver.cc:460] Webserver started at http://127.0.97.3:39211/ using document root <none> and password file <none>
I20260929 15:45:10.824119 7700 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:10.824380 7700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:10.831472 7700 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:45:10.836313 7715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:10.838842 7700 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 15:45:10.839121 7700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "3c17202443ee4dcdb74528f25fd6578f"
format_stamp: "Formatted at 2026-09-29 15:45:06 on dist-test-slave-7491"
server_key: "4febf9d037a0fbbe4983ddd85997c97a"
server_key_iv: "0a2f2057c890e33e72ce6e501429afa2"
server_key_version: "encryptionkey@0"
I20260929 15:45:10.840312 7700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:10.862440 7700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:10.869024 7700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:10.869408 7700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:10.870615 7700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:10.872289 7700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:10.872431 7700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:10.872576 7700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:10.872668 7700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:10.915553 7700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34081
I20260929 15:45:10.915606 7827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34081 every 8 connection(s)
I20260929 15:45:10.917624 7700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:10.925297 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 7700
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 15:45:10 dist-test-slave-7491 krb5kdc[6521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 15:45:10.967101 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.921773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44435 (local address 127.0.97.62:38733)
0929 15:45:10.922499 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.922508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.928860 (+ 6352us) server_negotiation.cc:407] Connection header received
0929 15:45:10.936224 (+ 7364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.936235 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.936380 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.936581 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.944685 (+ 8104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.945689 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.949507 (+ 3818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.949987 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.950149 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.956905 (+ 6756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.956914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.956920 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.956967 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.962354 (+ 5387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.963907 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.963915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.963919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.964102 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.964749 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.964755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.964776 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.965158 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.965308 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.966394 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.966660 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":332,"threads_started":1}
I20260929 15:45:10.971621 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.922498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:44435)
0929 15:45:10.923581 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:10.923644 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:10.935374 (+ 11730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:10.939913 (+ 4539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:10.939939 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:10.942472 (+ 2533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:10.944441 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.944476 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.945945 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.945950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.949243 (+ 3293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.949285 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.950272 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.950281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.950713 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.952027 (+ 1314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:10.952075 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:10.956490 (+ 4415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:10.962695 (+ 6205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.962708 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.962732 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.963704 (+ 972us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.964326 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.964332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.964336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.964611 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.965414 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:10.965424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:10.965926 (+ 502us) client_negotiation.cc:769] Sending connection context
0929 15:45:10.967824 (+ 1898us) client_negotiation.cc:240] Negotiation successful
0929 15:45:10.968524 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":7,"spinlock_wait_cycles":23808,"thread_start_us":481,"threads_started":1}
I20260929 15:45:10.973021 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.922903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.97.3:44103)
0929 15:45:10.938132 (+ 15229us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:10.938178 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:10.938378 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:10.940262 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:10.940272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:10.942161 (+ 1889us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:10.944440 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.944477 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.946184 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.946192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.949955 (+ 3763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.949981 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.953263 (+ 3282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.953271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.953461 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.954259 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:10.954282 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:10.959442 (+ 5160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:10.967399 (+ 7957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.967404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.967408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.968069 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.969736 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.969741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.969745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.969933 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.970816 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:10.970821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:10.971006 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 15:45:10.971195 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 15:45:10.971545 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14997,"mutex_wait_us":136,"thread_start_us":15589,"threads_started":1}
I20260929 15:45:10.973661 7829 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38733
I20260929 15:45:10.974028 7829 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:10.974253 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.922980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44103 (local address 127.0.97.61:38801)
0929 15:45:10.939035 (+ 16055us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.939039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.939094 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:45:10.939163 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.939168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.939248 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.939376 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.944747 (+ 5371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.945984 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.950234 (+ 4250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.951229 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.952982 (+ 1753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.959942 (+ 6960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.959955 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.960015 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.960088 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.967186 (+ 7098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.969323 (+ 2137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.969332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.969360 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.969570 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.970081 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.970086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.970090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.970579 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.970731 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.973537 (+ 2806us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.973847 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15942,"thread_start_us":16525,"threads_started":1}
I20260929 15:45:10.974944 7829 heartbeater.cc:511] Master 127.0.97.62:38733 requested a full tablet report, sending...
I20260929 15:45:10.977829 6565 ts_manager.cc:194] Re-registered known tserver with Master: 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081)
I20260929 15:45:10.979265 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44435
I20260929 15:45:10.980017 7830 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38801
I20260929 15:45:10.980257 7830 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:10.980880 7830 heartbeater.cc:511] Master 127.0.97.61:38801 requested a full tablet report, sending...
I20260929 15:45:10.984223 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.924971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.97.3:42839)
0929 15:45:10.941318 (+ 16347us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:10.941352 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:10.941998 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:10.943162 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:10.943167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:10.944359 (+ 1192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:10.944834 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.944848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.946320 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.946324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.952711 (+ 6387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:10.952743 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.954106 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.954117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:10.955345 (+ 1228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.956233 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:10.956264 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:10.969062 (+ 12798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:10.978180 (+ 9118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.978187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.978191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.978771 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.979276 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:10.979282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:10.979285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:10.979455 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:10.980568 (+ 1113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:10.980574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:10.983174 (+ 2600us) client_negotiation.cc:769] Sending connection context
0929 15:45:10.983331 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 15:45:10.983747 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16131,"thread_start_us":15889,"threads_started":1}
I20260929 15:45:10.985083 7168 ts_manager.cc:194] Registered new tserver with Master: 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081)
I20260929 15:45:10.986523 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.924504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42839 (local address 127.0.97.60:40467)
0929 15:45:10.942541 (+ 18037us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.942550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.942574 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:45:10.942649 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.942656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.942758 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.943004 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.945025 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.946141 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.952919 (+ 6778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.953592 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.953761 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.971842 (+ 18081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.971852 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.971857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.971900 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.977956 (+ 6056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.978940 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.978946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.978950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.979121 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.979601 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.979610 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.979616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.980328 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.980493 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.984722 (+ 4229us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.985031 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17893,"thread_start_us":17762,"threads_started":1}
I20260929 15:45:10.989585 7828 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40467
I20260929 15:45:10.989823 7828 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:10.990509 7828 heartbeater.cc:511] Master 127.0.97.60:40467 requested a full tablet report, sending...
I20260929 15:45:10.992419 7360 ts_manager.cc:194] Registered new tserver with Master: 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081)
I20260929 15:45:10.998112 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.947543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.0.97.62:38733)
0929 15:45:10.967395 (+ 19852us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.967402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.967421 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:45:10.967470 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.967475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.967551 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.967656 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.970085 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.971275 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.981671 (+ 10396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.982682 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.982819 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.988676 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.988684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.988689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.988744 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.993646 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.995184 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.995191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.995194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.995403 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.996042 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.996049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.996053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.996600 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.996906 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:10.997357 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 15:45:10.997671 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19752,"thread_start_us":46240,"threads_started":1}
I20260929 15:45:11.001300 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.947099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.0.97.61:38801)
0929 15:45:10.970255 (+ 23156us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.970260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.970278 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:45:10.970326 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.970330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.970397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.970549 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.981387 (+ 10838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.982594 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.984728 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.985476 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:10.985616 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:10.992031 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:10.992038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:10.992044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:10.992091 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:10.996845 (+ 4754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.997837 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.997842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.997846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.998057 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:10.998501 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:10.998506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:10.998509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:10.999235 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:10.999642 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.000461 (+ 819us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.000750 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22965,"thread_start_us":20453,"threads_started":1}
I20260929 15:45:11.004083 6565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:11.013440 6565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:11.013778 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:10.978074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.97.60:40467)
0929 15:45:10.986815 (+ 8741us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:10.986822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:10.994653 (+ 7831us) server_negotiation.cc:407] Connection header received
0929 15:45:10.994854 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:10.994860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:10.994996 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:10.995135 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:10.997717 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:10.998786 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.001257 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.002030 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.002171 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.007218 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:11.007225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:11.007229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:11.007283 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:11.010197 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.011224 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.011229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.011231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.011408 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.011866 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.011871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.011874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.012318 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:11.012588 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.012984 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.013229 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8610,"thread_start_us":10367,"threads_started":1}
I20260929 15:45:11.052192 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.041591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.97.2:45411)
0929 15:45:11.042259 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.042273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.042828 (+ 555us) server_negotiation.cc:407] Connection header received
0929 15:45:11.043421 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.043445 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.043859 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.044159 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:11.044866 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.046580 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.049229 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.050519 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.050703 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.050949 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.051468 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.051753 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":356,"threads_started":1}
I20260929 15:45:11.051997 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.041204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45411 (local address 127.0.0.1:60720)
0929 15:45:11.042691 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:11.042707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:11.042897 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:11.044055 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:11.044059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:11.044081 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:11.044519 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.044531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.046873 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.046878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.048847 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.048879 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.050747 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.050754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.050938 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.051072 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 15:45:11.051382 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 15:45:11.051550 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"threads_started":1}
I20260929 15:45:11.061972 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.042122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34081 (local address 127.0.0.1:48558)
0929 15:45:11.043727 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:11.043746 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:11.043854 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:11.052208 (+ 8354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:11.052215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:11.052252 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:11.052773 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.052788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.055600 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.055607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.058047 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.058064 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.060899 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.060907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.061103 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.061141 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:45:11.061271 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 15:45:11.061456 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"thread_start_us":651,"threads_started":1}
I20260929 15:45:11.062150 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.042895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.97.3:34081)
0929 15:45:11.051188 (+ 8293us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.051200 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.051244 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:45:11.051368 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.051376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.051738 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.051993 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:11.053116 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.055212 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.058306 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.060565 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.060779 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.061111 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.061360 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.061645 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5764,"thread_start_us":5555,"threads_started":1}
I20260929 15:45:11.066710 7763 tablet_service.cc:1467] Processing CreateTablet for tablet 65233e9ecf09419b9cab2d91b2b94c7e (DEFAULT_TABLE table=test-table [id=87f8c4384dfb4380bd02ed505c434686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:11.067683 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.043186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:33922)
0929 15:45:11.053387 (+ 10201us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:11.053405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:11.053608 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:11.061503 (+ 7895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:11.061507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:11.061559 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:11.061769 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.061782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.064024 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.064030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.065360 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.065373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.066788 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.066795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.067017 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.067071 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 15:45:11.067199 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:45:11.067354 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10065,"thread_start_us":672,"threads_started":1}
I20260929 15:45:11.067814 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.043653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.0.97.1:45041)
0929 15:45:11.060747 (+ 17094us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.060758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.060792 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:45:11.060907 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.060916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.061253 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.061967 (+ 714us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:11.062206 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.063860 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.065573 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.066553 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.066673 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.066846 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.067284 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.067467 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16874,"thread_start_us":16763,"threads_started":1}
I20260929 15:45:11.068773 7763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65233e9ecf09419b9cab2d91b2b94c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:11.071789 7473 tablet_service.cc:1467] Processing CreateTablet for tablet 65233e9ecf09419b9cab2d91b2b94c7e (DEFAULT_TABLE table=test-table [id=87f8c4384dfb4380bd02ed505c434686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:11.071789 7621 tablet_service.cc:1467] Processing CreateTablet for tablet 65233e9ecf09419b9cab2d91b2b94c7e (DEFAULT_TABLE table=test-table [id=87f8c4384dfb4380bd02ed505c434686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:11.073504 7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65233e9ecf09419b9cab2d91b2b94c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:11.073505 7621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65233e9ecf09419b9cab2d91b2b94c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:11.085896 7864 tablet_bootstrap.cc:492] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Bootstrap starting.
I20260929 15:45:11.089963 7864 tablet_bootstrap.cc:654] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:11.091207 7864 log.cc:824] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:11.095482 7865 tablet_bootstrap.cc:492] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Bootstrap starting.
I20260929 15:45:11.096642 7866 tablet_bootstrap.cc:492] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Bootstrap starting.
I20260929 15:45:11.096963 7864 tablet_bootstrap.cc:492] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: No bootstrap required, opened a new log
I20260929 15:45:11.097288 7864 ts_tablet_manager.cc:1397] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 15:45:11.099610 7865 tablet_bootstrap.cc:654] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:11.101223 7866 tablet_bootstrap.cc:654] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:11.101284 7865 log.cc:824] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:11.102300 7866 log.cc:824] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:11.105259 7865 tablet_bootstrap.cc:492] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: No bootstrap required, opened a new log
I20260929 15:45:11.105592 7865 ts_tablet_manager.cc:1397] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 15:45:11.105767 7866 tablet_bootstrap.cc:492] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: No bootstrap required, opened a new log
I20260929 15:45:11.106045 7866 ts_tablet_manager.cc:1397] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 15:45:11.108856 7864 raft_consensus.cc:348] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.109290 7864 raft_consensus.cc:374] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:11.109431 7864 raft_consensus.cc:729] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c17202443ee4dcdb74528f25fd6578f, State: Initialized, Role: FOLLOWER
I20260929 15:45:11.110136 7864 consensus_queue.cc:260] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.112139 7864 ts_tablet_manager.cc:1428] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260929 15:45:11.112535 7829 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
I20260929 15:45:11.118561 7865 raft_consensus.cc:348] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.118784 7866 raft_consensus.cc:348] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.119182 7866 raft_consensus.cc:374] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:11.119345 7865 raft_consensus.cc:374] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:11.119356 7866 raft_consensus.cc:729] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77332a715cbc4e62ac3edf365ad3b939, State: Initialized, Role: FOLLOWER
I20260929 15:45:11.119503 7865 raft_consensus.cc:729] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0b7a360cb482e8388398c020630c8, State: Initialized, Role: FOLLOWER
I20260929 15:45:11.120153 7866 consensus_queue.cc:260] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.120311 7865 consensus_queue.cc:260] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.122395 7865 ts_tablet_manager.cc:1428] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260929 15:45:11.122395 7866 ts_tablet_manager.cc:1428] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260929 15:45:11.123365 7688 heartbeater.cc:503] Master 127.0.97.62:38733 was elected leader, sending a full tablet report...
W20260929 15:45:11.128255 7541 tablet.cc:2369] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:11.137773 7872 raft_consensus.cc:482] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:11.138243 7872 raft_consensus.cc:504] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.140429 7872 leader_election.cc:288] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081), 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041)
I20260929 15:45:11.152895 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.140916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45057 (local address 127.0.97.3:34081)
0929 15:45:11.141141 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.141146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.142352 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 15:45:11.142590 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.142603 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.142729 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.142847 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:11.144689 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.146053 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.149905 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.151338 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.151525 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.151648 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.152420 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.152598 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:45:11.152895 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.142874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60599 (local address 127.0.97.1:45041)
0929 15:45:11.143135 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.143141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.143477 (+ 336us) server_negotiation.cc:407] Connection header received
0929 15:45:11.143749 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.143754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.143824 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.143930 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:11.144953 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.146160 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.149689 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.151338 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.151526 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.151648 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.152420 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.152589 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 15:45:11.153090 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.141006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34081 (local address 127.0.97.2:45057)
0929 15:45:11.142187 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:11.142213 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:11.142432 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:11.143270 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:11.143276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:11.143309 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:11.144383 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.144402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.146278 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.146286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.149413 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.149439 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.151719 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.151724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.152052 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.152212 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 15:45:11.152332 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 15:45:11.152573 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":341,"spinlock_wait_cycles":56064,"thread_start_us":651,"threads_started":1}
I20260929 15:45:11.153506 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.142571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.97.2:60599)
0929 15:45:11.143339 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:11.143357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:11.143565 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:11.143944 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:11.143948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:11.143970 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:11.144742 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.144756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.146299 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.146303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.149413 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:11.149439 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.151715 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.151722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:11.152050 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.152211 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 15:45:11.152332 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 15:45:11.152573 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":337,"thread_start_us":451,"threads_started":1}
I20260929 15:45:11.154660 7783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65233e9ecf09419b9cab2d91b2b94c7e" candidate_uuid: "09a0b7a360cb482e8388398c020630c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c17202443ee4dcdb74528f25fd6578f" is_pre_election: true
I20260929 15:45:11.155424 7783 raft_consensus.cc:2454] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a0b7a360cb482e8388398c020630c8 in term 0.
I20260929 15:45:11.155344 7493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65233e9ecf09419b9cab2d91b2b94c7e" candidate_uuid: "09a0b7a360cb482e8388398c020630c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77332a715cbc4e62ac3edf365ad3b939" is_pre_election: true
I20260929 15:45:11.156014 7493 raft_consensus.cc:2454] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a0b7a360cb482e8388398c020630c8 in term 0.
I20260929 15:45:11.156587 7575 leader_election.cc:302] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0b7a360cb482e8388398c020630c8, 3c17202443ee4dcdb74528f25fd6578f; no voters:
I20260929 15:45:11.157171 7872 raft_consensus.cc:2781] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:11.157352 7872 raft_consensus.cc:482] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:11.157477 7872 raft_consensus.cc:3037] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:45:11.162246 7690 tablet.cc:2369] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:11.162339 7872 raft_consensus.cc:504] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:11.163323 7872 leader_election.cc:288] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [CANDIDATE]: Term 1 election: Requested vote from peers 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081), 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041)
I20260929 15:45:11.163964 7493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65233e9ecf09419b9cab2d91b2b94c7e" candidate_uuid: "09a0b7a360cb482e8388398c020630c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77332a715cbc4e62ac3edf365ad3b939"
I20260929 15:45:11.163998 7783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65233e9ecf09419b9cab2d91b2b94c7e" candidate_uuid: "09a0b7a360cb482e8388398c020630c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c17202443ee4dcdb74528f25fd6578f"
I20260929 15:45:11.164251 7493 raft_consensus.cc:3037] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:11.164304 7783 raft_consensus.cc:3037] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:11.166807 7493 raft_consensus.cc:2454] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a0b7a360cb482e8388398c020630c8 in term 1.
I20260929 15:45:11.167475 7575 leader_election.cc:302] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0b7a360cb482e8388398c020630c8, 77332a715cbc4e62ac3edf365ad3b939; no voters:
I20260929 15:45:11.167740 7783 raft_consensus.cc:2454] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a0b7a360cb482e8388398c020630c8 in term 1.
I20260929 15:45:11.167868 7872 raft_consensus.cc:2781] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:11.168725 7872 raft_consensus.cc:686] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 1 LEADER]: Becoming Leader. State: Replica: 09a0b7a360cb482e8388398c020630c8, State: Running, Role: LEADER
I20260929 15:45:11.169327 7872 consensus_queue.cc:237] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
W20260929 15:45:11.171344 7831 tablet.cc:2369] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:11.175463 6565 catalog_manager.cc:5495] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a0b7a360cb482e8388398c020630c8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "09a0b7a360cb482e8388398c020630c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } health_report { overall_health: UNKNOWN } } }
I20260929 15:45:11.220150 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.213926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.97.2:45411)
0929 15:45:11.214155 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.214163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.214184 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:45:11.214293 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.214296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.214378 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.214550 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:11.214962 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.215700 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.217165 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.217584 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.217749 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.218287 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:11.219173 (+ 886us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:11.219494 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.219646 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.219765 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:45:11.231391 7783 raft_consensus.cc:1261] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Refusing update from remote peer 09a0b7a360cb482e8388398c020630c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:11.231446 7493 raft_consensus.cc:1261] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Refusing update from remote peer 09a0b7a360cb482e8388398c020630c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:11.232719 7875 consensus_queue.cc:1035] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:11.233157 7872 consensus_queue.cc:1035] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:11.278522 6565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:11.278919 6565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:11.288713 6565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdfc10a7ac2c4cfebb70413ae3961488: Sending DeleteTablet for 3 replicas of tablet 65233e9ecf09419b9cab2d91b2b94c7e
I20260929 15:45:11.289949 7621 tablet_service.cc:1514] Processing DeleteTablet for tablet 65233e9ecf09419b9cab2d91b2b94c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60720
I20260929 15:45:11.290371 7473 tablet_service.cc:1514] Processing DeleteTablet for tablet 65233e9ecf09419b9cab2d91b2b94c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33922
I20260929 15:45:11.290374 7763 tablet_service.cc:1514] Processing DeleteTablet for tablet 65233e9ecf09419b9cab2d91b2b94c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48558
I20260929 15:45:11.302958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 6534
I20260929 15:45:11.305231 6596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:11.308583 7889 tablet_replica.cc:330] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: stopping tablet replica
I20260929 15:45:11.309300 7889 raft_consensus.cc:2229] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:11.310565 7889 raft_consensus.cc:2258] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:11.314990 7889 ts_tablet_manager.cc:1905] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:11.322322 7891 tablet_replica.cc:330] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: stopping tablet replica
I20260929 15:45:11.322746 7890 tablet_replica.cc:330] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: stopping tablet replica
I20260929 15:45:11.323617 7890 raft_consensus.cc:2229] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:11.324463 7890 raft_consensus.cc:2258] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:11.326433 7891 raft_consensus.cc:2229] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:11.327533 7890 ts_tablet_manager.cc:1905] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:11.328647 7891 raft_consensus.cc:2258] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:11.331425 7891 ts_tablet_manager.cc:1905] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:11.328377 7889 ts_tablet_manager.cc:1918] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:11.335407 7889 log.cc:1196] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/65233e9ecf09419b9cab2d91b2b94c7e
I20260929 15:45:11.336406 7889 ts_tablet_manager.cc:1939] T 65233e9ecf09419b9cab2d91b2b94c7e P 09a0b7a360cb482e8388398c020630c8: Deleting consensus metadata
I20260929 15:45:11.341357 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.293847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.61:38801)
0929 15:45:11.294074 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.294083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.302454 (+ 8371us) server_negotiation.cc:407] Connection header received
0929 15:45:11.302899 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.302909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.303080 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.303187 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:11.316531 (+ 13344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.317728 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.323171 (+ 5443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.323933 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.324125 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.331041 (+ 6916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:11.331049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:11.331055 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:11.331105 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:11.336320 (+ 5215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.337662 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.337668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.337671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.337845 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.338317 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.338322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.338325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.338741 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:11.339258 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.340458 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.340930 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 15:45:11.347900 7890 ts_tablet_manager.cc:1918] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:11.348291 7890 log.cc:1196] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/65233e9ecf09419b9cab2d91b2b94c7e
I20260929 15:45:11.349018 7890 ts_tablet_manager.cc:1939] T 65233e9ecf09419b9cab2d91b2b94c7e P 77332a715cbc4e62ac3edf365ad3b939: Deleting consensus metadata
I20260929 15:45:11.352878 7891 ts_tablet_manager.cc:1918] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:11.353169 7891 log.cc:1196] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/65233e9ecf09419b9cab2d91b2b94c7e
I20260929 15:45:11.353843 7891 ts_tablet_manager.cc:1939] T 65233e9ecf09419b9cab2d91b2b94c7e P 3c17202443ee4dcdb74528f25fd6578f: Deleting consensus metadata
I20260929 15:45:11.366854 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.301592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.0.97.60:40467)
0929 15:45:11.301952 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.301981 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.345094 (+ 43113us) server_negotiation.cc:407] Connection header received
0929 15:45:11.345384 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.345390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.345483 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.345559 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:11.347750 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.349005 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.351206 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.351871 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.351997 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.357105 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:11.357115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:11.357121 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:11.357186 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:11.361796 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.363142 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.363150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.363155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.363489 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.364256 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.364263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.364267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.364690 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:11.364991 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.365909 (+ 918us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.366246 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 15:45:11.480943 6552 catalog_manager.cc:4841] TS 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411): tablet 65233e9ecf09419b9cab2d91b2b94c7e (table test-table [id=87f8c4384dfb4380bd02ed505c434686]) successfully deleted
I20260929 15:45:11.481118 6551 catalog_manager.cc:4841] TS 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081): tablet 65233e9ecf09419b9cab2d91b2b94c7e (table test-table [id=87f8c4384dfb4380bd02ed505c434686]) successfully deleted
I20260929 15:45:11.481459 6551 catalog_manager.cc:4841] TS 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041): tablet 65233e9ecf09419b9cab2d91b2b94c7e (table test-table [id=87f8c4384dfb4380bd02ed505c434686]) successfully deleted
I20260929 15:45:11.481966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 6534
I20260929 15:45:11.533605 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.514842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.0.97.60:40467)
0929 15:45:11.515248 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.515255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.515399 (+ 144us) server_negotiation.cc:407] Connection header received
0929 15:45:11.515664 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.515671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.515752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.515872 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:11.519043 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.519962 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.522913 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.523381 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.523540 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.526627 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:11.526636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:11.526648 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:11.526686 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:11.530290 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.531108 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.531115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.531119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.531270 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.531788 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.531793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.531797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.532111 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:11.532444 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.532966 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.533236 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:45:11.535609 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:11.512105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.0.97.61:38801)
0929 15:45:11.512461 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:11.512468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:11.513568 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 15:45:11.513739 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:11.513760 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:11.513900 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:11.514112 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:11.516461 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.518465 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.520675 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:11.521587 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:11.521895 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:11.525766 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:11.525775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:11.525782 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:11.525839 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:11.529897 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.530901 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.530908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.530912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.531080 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:11.532531 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:11.532536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:11.532540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:11.533044 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:11.533448 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:11.535133 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0929 15:45:11.535337 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260929 15:45:12.269634 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.268567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:39573)
0929 15:45:12.269179 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.269336 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":338,"threads_started":1}
I20260929 15:45:12.269630 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.268456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:42839)
0929 15:45:12.269083 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.269266 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":379,"threads_started":1}
W20260929 15:45:12.270321 7829 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
W20260929 15:45:12.270686 7688 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.272122 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.270812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:40517)
0929 15:45:12.271765 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.271924 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":726,"threads_started":1}
W20260929 15:45:12.272572 7539 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.788029 7903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdfc10a7ac2c4cfebb70413ae3961488)
I20260929 15:45:12.788511 7903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:12.789099 7904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdfc10a7ac2c4cfebb70413ae3961488)
I20260929 15:45:12.789428 7904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:12.790802 7903 leader_election.cc:288] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733), 39f9b121986f4071b281bdab26c1c316 (127.0.97.60:40467)
I20260929 15:45:12.792244 7904 leader_election.cc:288] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733), 115035d166124ced826936fab2b13803 (127.0.97.61:38801)
I20260929 15:45:12.792732 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.791210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52830)
0929 15:45:12.791974 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.792330 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":409,"threads_started":1}
W20260929 15:45:12.793284 7343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.793596 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.790752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52820)
0929 15:45:12.793131 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.793311 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2219,"thread_start_us":2049,"threads_started":1}
W20260929 15:45:12.794587 7152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.798195 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.796871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52844)
0929 15:45:12.797231 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.797350 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
W20260929 15:45:12.798501 7343 leader_election.cc:334] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.799108 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.797989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52848)
0929 15:45:12.798621 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.798803 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468}
W20260929 15:45:12.799536 7152 leader_election.cc:334] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.805508 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.792631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38801 (local address 127.0.0.1:39656)
0929 15:45:12.795015 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.795043 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:12.795185 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:12.797546 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:12.797550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:12.797570 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:12.798285 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:12.798302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.800726 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.800733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:12.803043 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:12.803057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.804252 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.804258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:12.804551 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:12.804602 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:45:12.804733 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 15:45:12.804903 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"mutex_wait_us":70,"thread_start_us":1960,"threads_started":1}
I20260929 15:45:12.805547 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.793485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.97.61:38801)
0929 15:45:12.797137 (+ 3652us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:12.797145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:12.797171 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:45:12.797257 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:12.797262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:12.797357 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:12.798224 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:12.799060 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.800485 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.803201 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.804007 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.804174 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:12.804328 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:12.804986 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 15:45:12.805147 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3519,"thread_start_us":3157,"threads_started":1}
I20260929 15:45:12.805770 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.793262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40467 (local address 127.0.0.1:37418)
0929 15:45:12.793820 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.793839 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:12.794000 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:12.798627 (+ 4627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:12.798633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:12.798668 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:12.798973 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:12.798985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.800740 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.800745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:12.802819 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:12.802840 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.804800 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.804806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:12.804993 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:12.805041 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 15:45:12.805162 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 15:45:12.805353 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":1010,"threads_started":1}
I20260929 15:45:12.806687 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.795042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.0.97.60:40467)
0929 15:45:12.797384 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:12.797391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:12.797417 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:45:12.797480 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:12.797486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:12.797556 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:12.798552 (+ 996us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:12.799198 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.800385 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.803057 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:12.804595 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:12.804744 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:12.804877 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:12.806046 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0929 15:45:12.806213 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":2172,"thread_start_us":1984,"threads_started":1}
I20260929 15:45:12.807401 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115035d166124ced826936fab2b13803" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "39f9b121986f4071b281bdab26c1c316" is_pre_election: true
I20260929 15:45:12.807582 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f9b121986f4071b281bdab26c1c316" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "115035d166124ced826936fab2b13803" is_pre_election: true
I20260929 15:45:12.807758 7370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115035d166124ced826936fab2b13803 in term 1.
I20260929 15:45:12.807930 7178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f9b121986f4071b281bdab26c1c316 in term 1.
I20260929 15:45:12.808583 7152 leader_election.cc:302] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 115035d166124ced826936fab2b13803, 39f9b121986f4071b281bdab26c1c316; no voters: bdfc10a7ac2c4cfebb70413ae3961488
I20260929 15:45:12.808583 7346 leader_election.cc:302] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 115035d166124ced826936fab2b13803, 39f9b121986f4071b281bdab26c1c316; no voters: bdfc10a7ac2c4cfebb70413ae3961488
I20260929 15:45:12.809031 7904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 15:45:12.809173 7904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bdfc10a7ac2c4cfebb70413ae3961488)
I20260929 15:45:12.809298 7904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:45:12.809876 7903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 15:45:12.810012 7903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bdfc10a7ac2c4cfebb70413ae3961488)
I20260929 15:45:12.810098 7903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:45:12.812996 7904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:12.813874 7904 leader_election.cc:288] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [CANDIDATE]: Term 2 election: Requested vote from peers bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733), 115035d166124ced826936fab2b13803 (127.0.97.61:38801)
I20260929 15:45:12.814378 7178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39f9b121986f4071b281bdab26c1c316" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "115035d166124ced826936fab2b13803"
I20260929 15:45:12.814495 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.814017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52866)
0929 15:45:12.814229 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.814336 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 15:45:12.814540 7903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:12.815521 7178 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f9b121986f4071b281bdab26c1c316 in current term 2: Already voted for candidate 115035d166124ced826936fab2b13803 in this term.
I20260929 15:45:12.815884 7903 leader_election.cc:288] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 2 election: Requested vote from peers bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733), 39f9b121986f4071b281bdab26c1c316 (127.0.97.60:40467)
I20260929 15:45:12.816875 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.816012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52884)
0929 15:45:12.816404 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.816511 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273}
I20260929 15:45:12.817195 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.815891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52882)
0929 15:45:12.816776 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.816912 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629}
W20260929 15:45:12.817444 7343 leader_election.cc:334] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.817637 7343 leader_election.cc:302] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f9b121986f4071b281bdab26c1c316; no voters: 115035d166124ced826936fab2b13803, bdfc10a7ac2c4cfebb70413ae3961488
I20260929 15:45:12.818004 7904 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 15:45:12.818007 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115035d166124ced826936fab2b13803" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "39f9b121986f4071b281bdab26c1c316"
I20260929 15:45:12.818377 7370 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 115035d166124ced826936fab2b13803 in current term 2: Already voted for candidate 39f9b121986f4071b281bdab26c1c316 in this term.
I20260929 15:45:12.819242 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:12.818801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:52898)
0929 15:45:12.818966 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:12.819095 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 15:45:12.819404 7152 leader_election.cc:334] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:12.819523 7152 leader_election.cc:302] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 115035d166124ced826936fab2b13803; no voters: 39f9b121986f4071b281bdab26c1c316, bdfc10a7ac2c4cfebb70413ae3961488
I20260929 15:45:12.819891 7903 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 15:45:13.272240 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:13.271437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:35249)
0929 15:45:13.271884 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:13.271998 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":260,"threads_started":1}
I20260929 15:45:13.272111 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:13.271197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:56411)
0929 15:45:13.271747 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:13.271855 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
I20260929 15:45:13.274253 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:13.273498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:47481)
0929 15:45:13.273984 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:13.274083 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":284,"threads_started":1}
I20260929 15:45:14.276103 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.273699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:50765)
0929 15:45:14.275537 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.275742 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1643,"thread_start_us":1518,"threads_started":1}
I20260929 15:45:14.276686 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.273699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:58383)
0929 15:45:14.276194 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.276307 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2300,"thread_start_us":2179,"threads_started":1}
I20260929 15:45:14.278092 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.276918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:41887)
0929 15:45:14.277722 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.277853 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":377,"threads_started":1}
I20260929 15:45:14.800432 7919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:14.800760 7919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:14.801815 7919 leader_election.cc:288] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733), 39f9b121986f4071b281bdab26c1c316 (127.0.97.60:40467)
I20260929 15:45:14.803082 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.802053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47236)
0929 15:45:14.802650 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.802805 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
I20260929 15:45:14.803053 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115035d166124ced826936fab2b13803" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "39f9b121986f4071b281bdab26c1c316" is_pre_election: true
I20260929 15:45:14.803536 7370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115035d166124ced826936fab2b13803 in term 2.
I20260929 15:45:14.804798 7152 leader_election.cc:302] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115035d166124ced826936fab2b13803, 39f9b121986f4071b281bdab26c1c316; no voters:
I20260929 15:45:14.805202 7919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 15:45:14.805330 7919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:14.805434 7919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:45:14.805828 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.805386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47248)
0929 15:45:14.805579 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.805675 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 15:45:14.806079 7152 leader_election.cc:334] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:14.808122 7919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:14.808907 7919 leader_election.cc:288] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 3 election: Requested vote from peers bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733), 39f9b121986f4071b281bdab26c1c316 (127.0.97.60:40467)
I20260929 15:45:14.809587 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.809057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47254)
0929 15:45:14.809245 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.809392 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 15:45:14.809783 7370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115035d166124ced826936fab2b13803" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "39f9b121986f4071b281bdab26c1c316"
I20260929 15:45:14.810036 7370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:45:14.810889 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.810520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47264)
0929 15:45:14.810662 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.810751 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 15:45:14.811046 7152 leader_election.cc:334] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
I20260929 15:45:14.812534 7370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 115035d166124ced826936fab2b13803 in term 3.
I20260929 15:45:14.813289 7152 leader_election.cc:302] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 115035d166124ced826936fab2b13803, 39f9b121986f4071b281bdab26c1c316; no voters: bdfc10a7ac2c4cfebb70413ae3961488
I20260929 15:45:14.813689 7919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 15:45:14.814257 7919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [term 3 LEADER]: Becoming Leader. State: Replica: 115035d166124ced826936fab2b13803, State: Running, Role: LEADER
I20260929 15:45:14.814682 7919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } }
I20260929 15:45:14.816823 7922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115035d166124ced826936fab2b13803. Latest consensus state: current_term: 3 leader_uuid: "115035d166124ced826936fab2b13803" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:14.817268 7922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:14.818084 7924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:14.822742 7924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:14.823928 7924 catalog_manager.cc:1259] Loaded cluster ID: 1e8830dd33ae4aafac80e4f49874a95c
I20260929 15:45:14.824011 7924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:14.825487 7924 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:14.826596 7924 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Loaded TSK: 0
I20260929 15:45:14.827772 7924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:45:14.870071 7167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:14.878309 7167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:14.884509 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.883842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47268)
0929 15:45:14.884208 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.884343 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 15:45:14.884606 7370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [term 3 FOLLOWER]: Refusing update from remote peer 115035d166124ced826936fab2b13803: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 15:45:14.885793 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.885344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47276)
0929 15:45:14.885509 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.885588 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 15:45:14.885792 7922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 15:45:14.886210 7152 consensus_peers.cc:487] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 -> Peer bdfc10a7ac2c4cfebb70413ae3961488 (127.0.97.62:38733): Couldn't send request to peer bdfc10a7ac2c4cfebb70413ae3961488. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:45:14.890476 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115035d166124ced826936fab2b13803. Latest consensus state: current_term: 3 leader_uuid: "115035d166124ced826936fab2b13803" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:14.890796 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:45:14.892388 7922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "115035d166124ced826936fab2b13803" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:14.892743 7922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:14.893872 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "115035d166124ced826936fab2b13803" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdfc10a7ac2c4cfebb70413ae3961488" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38733 } } peers { permanent_uuid: "115035d166124ced826936fab2b13803" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38801 } attrs { promote: false } } peers { permanent_uuid: "39f9b121986f4071b281bdab26c1c316" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40467 } attrs { promote: false } } }
I20260929 15:45:14.894290 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f9b121986f4071b281bdab26c1c316 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:45:14.912915 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.902696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.0.97.2:45411)
0929 15:45:14.903290 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:14.903297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:14.903320 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:45:14.903400 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:14.903406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:14.903494 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:14.904261 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:14.904463 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.905691 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.909400 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.911165 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.911364 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:14.911517 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:14.912277 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 15:45:14.912445 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":332,"threads_started":1}
I20260929 15:45:14.912907 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.902564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45411 (local address 127.0.0.1:33790)
0929 15:45:14.902788 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.902809 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:14.902940 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:14.903725 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:14.903735 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:14.903779 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:14.904315 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:14.904335 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.905869 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.905878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:14.909083 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:14.909115 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.911464 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.911477 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:14.911828 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:14.911896 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 15:45:14.912059 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 15:45:14.912336 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 15:45:14.914520 7621 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa4737b49f349df8f3da95b99aecce1 (DEFAULT_TABLE table=test-table [id=886f550fe36645c294231c0286247eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:14.915355 7621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa4737b49f349df8f3da95b99aecce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:14.932055 7937 tablet_bootstrap.cc:492] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Bootstrap starting.
I20260929 15:45:14.935027 7937 tablet_bootstrap.cc:654] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:14.936244 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.904585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46386)
0929 15:45:14.905219 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.905246 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:14.905399 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:14.925370 (+ 19971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:14.925379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:14.925430 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:14.926001 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:14.926018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.928288 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.928297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:14.931490 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:14.931521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.934735 (+ 3214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.934749 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:14.935092 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:14.935170 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 15:45:14.935292 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 15:45:14.935576 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":1218,"threads_started":1}
I20260929 15:45:14.938185 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.904966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.0.97.1:45041)
0929 15:45:14.924778 (+ 19812us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:14.924789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:14.924827 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:45:14.924944 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:14.924953 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:14.925071 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:14.926259 (+ 1188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:14.926483 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.928010 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.931888 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.934199 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.937087 (+ 2888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:14.937299 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:14.937409 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 15:45:14.937684 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19593,"thread_start_us":19452,"threads_started":1}
I20260929 15:45:14.941421 7473 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa4737b49f349df8f3da95b99aecce1 (DEFAULT_TABLE table=test-table [id=886f550fe36645c294231c0286247eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:14.942070 7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa4737b49f349df8f3da95b99aecce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:14.944032 7937 tablet_bootstrap.cc:492] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: No bootstrap required, opened a new log
I20260929 15:45:14.944310 7937 ts_tablet_manager.cc:1397] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 15:45:14.945453 7937 raft_consensus.cc:348] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:14.945776 7937 raft_consensus.cc:374] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:14.945940 7937 raft_consensus.cc:729] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0b7a360cb482e8388398c020630c8, State: Initialized, Role: FOLLOWER
I20260929 15:45:14.946666 7937 consensus_queue.cc:260] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:14.948302 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.902909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34081 (local address 127.0.0.1:43652)
0929 15:45:14.903526 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.903548 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:14.903676 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:14.939679 (+ 36003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:14.939687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:14.939735 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:14.940380 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:14.940396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.941941 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.941948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:14.944426 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:14.944461 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.947212 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.947219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:14.947473 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:14.947520 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 15:45:14.947636 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:45:14.947836 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":357,"threads_started":1}
I20260929 15:45:14.948624 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.903412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.0.97.3:34081)
0929 15:45:14.939207 (+ 35795us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:14.939212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:14.939237 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:45:14.939326 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:14.939331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:14.939442 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:14.939624 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:14.940634 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.941703 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.944762 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:14.946856 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:14.947123 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:14.947264 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:14.947712 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 15:45:14.947978 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35617,"thread_start_us":35396,"threads_started":1}
I20260929 15:45:14.949823 7763 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa4737b49f349df8f3da95b99aecce1 (DEFAULT_TABLE table=test-table [id=886f550fe36645c294231c0286247eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:14.950340 7940 tablet_bootstrap.cc:492] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: Bootstrap starting.
I20260929 15:45:14.950384 7763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa4737b49f349df8f3da95b99aecce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:14.953879 7937 ts_tablet_manager.cc:1428] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20260929 15:45:14.954072 7940 tablet_bootstrap.cc:654] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:14.958890 7689 heartbeater.cc:503] Master 127.0.97.61:38801 was elected leader, sending a full tablet report...
I20260929 15:45:14.964079 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.951119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:43445)
0929 15:45:14.963601 (+ 12482us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.963773 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12309,"threads_started":1}
I20260929 15:45:14.964421 7940 tablet_bootstrap.cc:492] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: No bootstrap required, opened a new log
I20260929 15:45:14.964596 7940 ts_tablet_manager.cc:1397] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260929 15:45:14.965634 7940 raft_consensus.cc:348] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:14.965891 7940 raft_consensus.cc:374] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:14.965991 7940 raft_consensus.cc:729] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77332a715cbc4e62ac3edf365ad3b939, State: Initialized, Role: FOLLOWER
I20260929 15:45:14.966038 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.965073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:47023)
0929 15:45:14.965745 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.965852 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"threads_started":1}
I20260929 15:45:14.966308 7940 consensus_queue.cc:260] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:14.967504 7943 tablet_bootstrap.cc:492] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Bootstrap starting.
I20260929 15:45:14.970420 7943 tablet_bootstrap.cc:654] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:14.971189 7940 ts_tablet_manager.cc:1428] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 15:45:14.973632 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.971281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:34149)
0929 15:45:14.973258 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.973396 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1860,"threads_started":1}
I20260929 15:45:14.975848 7943 tablet_bootstrap.cc:492] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: No bootstrap required, opened a new log
I20260929 15:45:14.976055 7943 ts_tablet_manager.cc:1397] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 15:45:14.976987 7943 raft_consensus.cc:348] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:14.977237 7943 raft_consensus.cc:374] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:14.977342 7943 raft_consensus.cc:729] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c17202443ee4dcdb74528f25fd6578f, State: Initialized, Role: FOLLOWER
I20260929 15:45:14.977794 7943 consensus_queue.cc:260] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:14.979050 7943 ts_tablet_manager.cc:1428] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:45:14.981510 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.980554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:45567)
0929 15:45:14.981209 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.981323 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509}
I20260929 15:45:14.982487 7540 heartbeater.cc:503] Master 127.0.97.61:38801 was elected leader, sending a full tablet report...
I20260929 15:45:14.984050 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.979450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:55453)
0929 15:45:14.983509 (+ 4059us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.983702 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3878,"threads_started":1}
I20260929 15:45:14.986503 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:14.985953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:53891)
0929 15:45:14.986203 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:14.986300 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 15:45:14.986562 7830 heartbeater.cc:503] Master 127.0.97.61:38801 was elected leader, sending a full tablet report...
I20260929 15:45:15.036652 7945 raft_consensus.cc:482] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:15.037009 7945 raft_consensus.cc:504] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:15.039448 7945 leader_election.cc:288] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411), 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081)
I20260929 15:45:15.047255 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.039690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45411 (local address 127.0.97.1:39171)
0929 15:45:15.039885 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.039910 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:15.040089 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:15.040800 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:15.040808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:15.040864 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:15.041165 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:15.041174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.042756 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.042762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:15.044695 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:15.044709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.046439 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.046444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:15.046592 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.046621 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:45:15.046719 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:45:15.046939 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 15:45:15.047252 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.039942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39171 (local address 127.0.97.2:45411)
0929 15:45:15.040293 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:15.040299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:15.040316 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:45:15.040392 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:15.040396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:15.040529 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:15.040631 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:15.041417 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.042571 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.044891 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.046186 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.046334 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.046458 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:15.046784 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 15:45:15.046937 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 15:45:15.048293 7641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa4737b49f349df8f3da95b99aecce1" candidate_uuid: "77332a715cbc4e62ac3edf365ad3b939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0b7a360cb482e8388398c020630c8" is_pre_election: true
I20260929 15:45:15.048507 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.040386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33095 (local address 127.0.97.3:34081)
0929 15:45:15.040700 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:15.040707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:15.041269 (+ 562us) server_negotiation.cc:407] Connection header received
0929 15:45:15.041564 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:15.041569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:15.041705 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:15.041808 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:15.042527 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.043702 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.046093 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.047281 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.047414 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.047527 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:15.048075 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 15:45:15.048256 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 15:45:15.048537 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.040066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34081 (local address 127.0.97.1:33095)
0929 15:45:15.041115 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.041140 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:15.041359 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:15.041877 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:15.041885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:15.041917 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:15.042288 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:15.042301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.043880 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.043884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:15.045819 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:15.045835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.047707 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.047713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:15.047898 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.047922 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:45:15.048039 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 15:45:15.048181 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":582,"threads_started":1}
I20260929 15:45:15.048816 7641 raft_consensus.cc:2454] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77332a715cbc4e62ac3edf365ad3b939 in term 0.
I20260929 15:45:15.049491 7783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa4737b49f349df8f3da95b99aecce1" candidate_uuid: "77332a715cbc4e62ac3edf365ad3b939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c17202443ee4dcdb74528f25fd6578f" is_pre_election: true
I20260929 15:45:15.049463 7428 leader_election.cc:302] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0b7a360cb482e8388398c020630c8, 77332a715cbc4e62ac3edf365ad3b939; no voters:
I20260929 15:45:15.049768 7783 raft_consensus.cc:2454] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77332a715cbc4e62ac3edf365ad3b939 in term 0.
I20260929 15:45:15.050066 7945 raft_consensus.cc:2781] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:15.050249 7945 raft_consensus.cc:482] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:15.050346 7945 raft_consensus.cc:3037] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:15.052990 7945 raft_consensus.cc:504] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:15.053656 7945 leader_election.cc:288] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [CANDIDATE]: Term 1 election: Requested vote from peers 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411), 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081)
I20260929 15:45:15.054361 7783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa4737b49f349df8f3da95b99aecce1" candidate_uuid: "77332a715cbc4e62ac3edf365ad3b939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c17202443ee4dcdb74528f25fd6578f"
I20260929 15:45:15.054406 7641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa4737b49f349df8f3da95b99aecce1" candidate_uuid: "77332a715cbc4e62ac3edf365ad3b939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0b7a360cb482e8388398c020630c8"
I20260929 15:45:15.054632 7783 raft_consensus.cc:3037] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:15.054644 7641 raft_consensus.cc:3037] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:15.057284 7641 raft_consensus.cc:2454] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77332a715cbc4e62ac3edf365ad3b939 in term 1.
I20260929 15:45:15.057758 7428 leader_election.cc:302] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0b7a360cb482e8388398c020630c8, 77332a715cbc4e62ac3edf365ad3b939; no voters:
I20260929 15:45:15.058092 7945 raft_consensus.cc:2781] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:15.058250 7783 raft_consensus.cc:2454] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77332a715cbc4e62ac3edf365ad3b939 in term 1.
I20260929 15:45:15.058683 7945 raft_consensus.cc:686] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 1 LEADER]: Becoming Leader. State: Replica: 77332a715cbc4e62ac3edf365ad3b939, State: Running, Role: LEADER
I20260929 15:45:15.059250 7945 consensus_queue.cc:237] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } }
I20260929 15:45:15.060638 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.059952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:54391)
0929 15:45:15.060247 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.060395 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:45:15.064869 7168 catalog_manager.cc:5495] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "77332a715cbc4e62ac3edf365ad3b939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77332a715cbc4e62ac3edf365ad3b939" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45041 } health_report { overall_health: HEALTHY } } }
I20260929 15:45:15.097957 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.091494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.0.97.1:45041)
0929 15:45:15.091725 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:15.091729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:15.091748 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:45:15.091844 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:15.091848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:15.091949 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:15.092102 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:15.092838 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.093718 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.095208 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.095656 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.095783 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.096408 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:15.097107 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:15.097246 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:15.097567 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 15:45:15.097653 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:45:15.103592 7783 raft_consensus.cc:1261] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Refusing update from remote peer 77332a715cbc4e62ac3edf365ad3b939: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:15.103966 7641 raft_consensus.cc:1261] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Refusing update from remote peer 77332a715cbc4e62ac3edf365ad3b939: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:15.104785 7951 consensus_queue.cc:1035] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c17202443ee4dcdb74528f25fd6578f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:15.105242 7945 consensus_queue.cc:1035] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a0b7a360cb482e8388398c020630c8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:15.108608 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.107828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:45207)
0929 15:45:15.108160 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.108297 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260929 15:45:15.110088 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.108882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:51143)
0929 15:45:15.109213 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.109479 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 15:45:15.114073 7954 mvcc.cc:205] Tried to move back new op lower bound from 7334693745055600640 to 7334693744889827328. Current Snapshot: MvccSnapshot[applied={T|T < 7334693745055600640}]
I20260929 15:45:15.114228 7955 mvcc.cc:205] Tried to move back new op lower bound from 7334693745055600640 to 7334693744889827328. Current Snapshot: MvccSnapshot[applied={T|T < 7334693745055600640}]
I20260929 15:45:15.115597 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.114949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.3:43431)
0929 15:45:15.115295 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.115409 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260929 15:45:15.117619 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.116723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.1:58907)
0929 15:45:15.117260 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.117411 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260929 15:45:15.123087 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.121803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.97.2:51801)
0929 15:45:15.122632 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.122738 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696}
I20260929 15:45:15.124133 7957 mvcc.cc:205] Tried to move back new op lower bound from 7334693745055600640 to 7334693744889827328. Current Snapshot: MvccSnapshot[applied={T|T < 7334693745055600640}]
I20260929 15:45:15.132697 7168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:15.133082 7168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:15.141633 7168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 115035d166124ced826936fab2b13803: Sending DeleteTablet for 3 replicas of tablet 4aa4737b49f349df8f3da95b99aecce1
I20260929 15:45:15.142858 7621 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa4737b49f349df8f3da95b99aecce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:15 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33790
I20260929 15:45:15.143355 7763 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa4737b49f349df8f3da95b99aecce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:15 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:43652
I20260929 15:45:15.144116 7473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa4737b49f349df8f3da95b99aecce1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:15 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46386
I20260929 15:45:15.144860 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 7345
I20260929 15:45:15.157286 7964 tablet_replica.cc:330] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: stopping tablet replica
I20260929 15:45:15.157521 7964 raft_consensus.cc:2229] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:15.157811 7964 raft_consensus.cc:2258] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:15.161794 7964 ts_tablet_manager.cc:1905] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:15.175652 7965 tablet_replica.cc:330] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: stopping tablet replica
I20260929 15:45:15.176426 7965 raft_consensus.cc:2229] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:15.177548 7965 raft_consensus.cc:2258] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:15.179275 7964 ts_tablet_manager.cc:1918] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:15.179432 7965 ts_tablet_manager.cc:1905] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:15.179611 7964 log.cc:1196] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/4aa4737b49f349df8f3da95b99aecce1
I20260929 15:45:15.189083 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.146580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.97.1:45041)
0929 15:45:15.146783 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:15.146789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:15.156786 (+ 9997us) server_negotiation.cc:407] Connection header received
0929 15:45:15.158362 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:15.158369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:15.158477 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:15.158550 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:15.164358 (+ 5808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.165640 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.167634 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.168454 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.168597 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.180437 (+ 11840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:15.180461 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:15.180492 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:15.180550 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:15.185182 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:15.186301 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:15.186312 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:15.186315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:15.186521 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:15.187010 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:15.187015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:15.187019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:15.187474 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:15.187780 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:15.188422 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 15:45:15.188678 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:45:15.190256 7964 ts_tablet_manager.cc:1939] T 4aa4737b49f349df8f3da95b99aecce1 P 3c17202443ee4dcdb74528f25fd6578f: Deleting consensus metadata
I20260929 15:45:15.190685 7533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:15.191134 7963 tablet_replica.cc:330] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: stopping tablet replica
I20260929 15:45:15.191603 7153 catalog_manager.cc:4841] TS 3c17202443ee4dcdb74528f25fd6578f (127.0.97.3:34081): tablet 4aa4737b49f349df8f3da95b99aecce1 (table test-table [id=886f550fe36645c294231c0286247eda]) successfully deleted
I20260929 15:45:15.191924 7963 raft_consensus.cc:2229] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:15.192322 7963 raft_consensus.cc:2258] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:15.195060 7963 ts_tablet_manager.cc:1905] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:15.202507 7963 ts_tablet_manager.cc:1918] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:15.202698 7963 log.cc:1196] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/4aa4737b49f349df8f3da95b99aecce1
I20260929 15:45:15.203195 7963 ts_tablet_manager.cc:1939] T 4aa4737b49f349df8f3da95b99aecce1 P 09a0b7a360cb482e8388398c020630c8: Deleting consensus metadata
I20260929 15:45:15.204685 7154 catalog_manager.cc:4841] TS 09a0b7a360cb482e8388398c020630c8 (127.0.97.2:45411): tablet 4aa4737b49f349df8f3da95b99aecce1 (table test-table [id=886f550fe36645c294231c0286247eda]) successfully deleted
I20260929 15:45:15.309382 7965 ts_tablet_manager.cc:1918] T 4aa4737b49f349df8f3da95b99aecce1 P 77332a715cbc4e62ac3edf365ad3b939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:15.309468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 7345
I20260929 15:45:15.331730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 7550
I20260929 15:45:15.332983 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.332501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46412)
0929 15:45:15.332736 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.332827 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 15:45:15.333293 7153 catalog_manager.cc:4584] TS 77332a715cbc4e62ac3edf365ad3b939 (127.0.97.1:45041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4aa4737b49f349df8f3da95b99aecce1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
I20260929 15:45:15.350216 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.332922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.97.2:45411)
0929 15:45:15.333592 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:15.333599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:15.333627 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:45:15.333695 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:15.333699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:15.333784 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:15.334036 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:15.335698 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.336706 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.338301 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.338832 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.339090 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.342334 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:15.342352 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:15.342369 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:15.342416 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:15.346319 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:15.347364 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:15.347372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:15.347374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:15.347543 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:15.348251 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:15.348255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:15.348257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:15.348800 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:15.349063 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:15.349612 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 15:45:15.349827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511}
I20260929 15:45:15.351022 7682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:15.386023 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.385522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46426)
0929 15:45:15.385756 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.385848 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 15:45:15.387163 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.386799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46428)
0929 15:45:15.386948 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.387031 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:45:15.457132 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.456651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46444)
0929 15:45:15.456884 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.456969 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 15:45:15.458750 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.458267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46446)
0929 15:45:15.458467 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.458576 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 15:45:15.483693 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.483137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47278)
0929 15:45:15.483402 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.483510 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 15:45:15.484928 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.484537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38733 (local address 127.0.0.1:47290)
0929 15:45:15.484698 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.484799 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 15:45:15.496937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 7550
I20260929 15:45:15.522428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 7700
I20260929 15:45:15.539242 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.523284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.97.3:34081)
0929 15:45:15.523450 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:15.523455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:15.523478 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:45:15.523742 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:15.523747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:15.523817 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:15.523916 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:15.525952 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.526709 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.528651 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:15.529091 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:15.529384 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:15.532129 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:15.532150 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:15.532261 (+ 111us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:15.532317 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:15.535291 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:15.536348 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:15.536362 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:15.536366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:15.536566 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:15.537312 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:15.537318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:15.537320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:15.537791 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:15.538068 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:15.538604 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 15:45:15.538875 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:45:15.540135 7823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:15.566783 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.566257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46448)
0929 15:45:15.566499 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.566600 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 15:45:15.568042 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.567625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46456)
0929 15:45:15.567822 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.567909 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 15:45:15.645895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 7700
I20260929 15:45:15.665766 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 7029
I20260929 15:45:15.667088 7198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:15.813228 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:15.812599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45041 (local address 127.0.0.1:46464)
0929 15:45:15.812861 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:15.812979 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 15:45:15.813690 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 7029
I20260929 15:45:15.839821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 7125
I20260929 15:45:15.841472 7390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:15.979296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 7125
2026-09-29T15:45:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (13135 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:45:16 dist-test-slave-7491 krb5kdc[7974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:45:16 dist-test-slave-7491 krb5kdc[7974](info): set up 2 sockets
Sep 29 15:45:16 dist-test-slave-7491 krb5kdc[7974](info): commencing operation
krb5kdc: starting...
W20260929 15:45:17.149708 388 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:45:17 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 15:45:17.420943 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38809
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:17.814631 7987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:17.815095 7987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:17.815294 7987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:17.823429 7987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:17.823659 7987 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:45:17.823751 7987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:17.823876 7987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:17.823975 7987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:17.824069 7987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:17.835299 7987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38809
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:17.837023 7987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:17.838549 7987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:17.853308 7992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:17.853338 7995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:17.853259 7993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:17.853706 7987 server_base.cc:1034] running on GCE node
I20260929 15:45:17.854960 7987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:17.856774 7987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:17.858081 7987 hybrid_clock.cc:648] HybridClock initialized: now 1790696717857922 us; error 140 us; skew 500 ppm
I20260929 15:45:17.858516 7987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:17 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:17.862298 7987 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 15:45:17.864106 7987 webserver.cc:460] Webserver started at http://127.0.97.62:36423/ using document root <none> and password file <none>
I20260929 15:45:17.864972 7987 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:17.865072 7987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:17.865378 7987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:17.868719 7987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "7674b1f0d843449f8e592dc3d8f53a06"
format_stamp: "Formatted at 2026-09-29 15:45:17 on dist-test-slave-7491"
server_key: "6f0f98de804f4ae25dfaab762342e4b5"
server_key_iv: "77aae15b4dacdb7b3d6e681c1bc5e956"
server_key_version: "encryptionkey@0"
I20260929 15:45:17.869642 7987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "7674b1f0d843449f8e592dc3d8f53a06"
format_stamp: "Formatted at 2026-09-29 15:45:17 on dist-test-slave-7491"
server_key: "6f0f98de804f4ae25dfaab762342e4b5"
server_key_iv: "77aae15b4dacdb7b3d6e681c1bc5e956"
server_key_version: "encryptionkey@0"
I20260929 15:45:17.875610 7987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:45:17.881186 8002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:17.883700 7987 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:45:17.883947 7987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "7674b1f0d843449f8e592dc3d8f53a06"
format_stamp: "Formatted at 2026-09-29 15:45:17 on dist-test-slave-7491"
server_key: "6f0f98de804f4ae25dfaab762342e4b5"
server_key_iv: "77aae15b4dacdb7b3d6e681c1bc5e956"
server_key_version: "encryptionkey@0"
I20260929 15:45:17.884388 7987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:17.925885 7987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:17.933095 7987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:17.933957 7987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:17.968254 7987 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38809
I20260929 15:45:17.968386 8063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38809 every 8 connection(s)
I20260929 15:45:17.969998 7987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:17.975438 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:17.979354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 7987
I20260929 15:45:17.979686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:17.980422 388 external_mini_cluster.cc:1428] Setting key 4525b2f4aa6560c877d0815c0968ce9f
I20260929 15:45:17.991748 8064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Bootstrap starting.
I20260929 15:45:17.996259 8064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:17.997800 8064 log.cc:824] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:45:17 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:18.001546 8064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: No bootstrap required, opened a new log
I20260929 15:45:18.012771 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:17.983320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.0.97.62:38809)
0929 15:45:17.984682 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:17.984699 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:17.984763 (+ 64us) server_negotiation.cc:407] Connection header received
0929 15:45:17.986900 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:17.986934 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:17.987472 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:17.988143 (+ 671us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:17.990173 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:17.991877 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:17.993918 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:17.995260 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:17.995528 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:18.001934 (+ 6406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:18.001947 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:18.001969 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:18.002023 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:18.006768 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:18.007850 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:18.007861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:18.007871 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:18.008126 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:18.008621 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:18.008628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:18.008633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:18.009573 (+ 940us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:18.009917 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:18.010432 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 15:45:18.011005 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":552,"threads_started":1}
I20260929 15:45:18.013672 8064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7674b1f0d843449f8e592dc3d8f53a06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260929 15:45:18.014150 8064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:18.014281 8064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7674b1f0d843449f8e592dc3d8f53a06, State: Initialized, Role: FOLLOWER
I20260929 15:45:18.015022 8064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7674b1f0d843449f8e592dc3d8f53a06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260929 15:45:18.015354 8064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:18.015503 8064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:18.015687 8064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:18.018167 8064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7674b1f0d843449f8e592dc3d8f53a06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260929 15:45:18.018844 8064 leader_election.cc:302] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7674b1f0d843449f8e592dc3d8f53a06; no voters:
I20260929 15:45:18.019760 8064 leader_election.cc:288] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:18.020053 8069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:18.021227 8069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [term 1 LEADER]: Becoming Leader. State: Replica: 7674b1f0d843449f8e592dc3d8f53a06, State: Running, Role: LEADER
I20260929 15:45:18.021944 8069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7674b1f0d843449f8e592dc3d8f53a06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260929 15:45:18.022326 8064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:18.025219 8071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7674b1f0d843449f8e592dc3d8f53a06. Latest consensus state: current_term: 1 leader_uuid: "7674b1f0d843449f8e592dc3d8f53a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7674b1f0d843449f8e592dc3d8f53a06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } } }
I20260929 15:45:18.025712 8071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:18.025552 8070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7674b1f0d843449f8e592dc3d8f53a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7674b1f0d843449f8e592dc3d8f53a06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } } }
I20260929 15:45:18.025876 8070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:18.028836 8078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:18.035560 8078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:18.047493 8078 catalog_manager.cc:1347] Generated new cluster ID: c0b3e60e4f1c4c61b56eddee38debc82
I20260929 15:45:18.047700 8078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:18.080035 8078 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:18.081624 8078 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:18.090853 8078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Generated new TSK 0
I20260929 15:45:18.092136 8078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 15:45:18.164728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:18.562228 8090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:18.562682 8090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:18.562803 8090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:18.571352 8090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:18.571504 8090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:18.571600 8090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:45:18.571707 8090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:18.584245 8090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:18.585968 8090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:18.587383 8090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:18.604395 8095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:18.604271 8096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:18.604964 8098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:18.605017 8090 server_base.cc:1034] running on GCE node
I20260929 15:45:18.605875 8090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:18.607301 8090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:18.608666 8090 hybrid_clock.cc:648] HybridClock initialized: now 1790696718608558 us; error 86 us; skew 500 ppm
I20260929 15:45:18.609061 8090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:18 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:18.612936 8090 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:45:18.614912 8090 webserver.cc:460] Webserver started at http://127.0.97.1:36509/ using document root <none> and password file <none>
I20260929 15:45:18.615681 8090 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:18.615787 8090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:18.616089 8090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:18.619915 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd"
format_stamp: "Formatted at 2026-09-29 15:45:18 on dist-test-slave-7491"
server_key: "703e2776bcc390ca82f7f8207528912b"
server_key_iv: "eb08f3a006fc10604f5766105c589e0b"
server_key_version: "encryptionkey@0"
I20260929 15:45:18.621353 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd"
format_stamp: "Formatted at 2026-09-29 15:45:18 on dist-test-slave-7491"
server_key: "703e2776bcc390ca82f7f8207528912b"
server_key_iv: "eb08f3a006fc10604f5766105c589e0b"
server_key_version: "encryptionkey@0"
I20260929 15:45:18.628870 8090 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260929 15:45:18.634114 8105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:18.636389 8090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:45:18.636665 8090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd"
format_stamp: "Formatted at 2026-09-29 15:45:18 on dist-test-slave-7491"
server_key: "703e2776bcc390ca82f7f8207528912b"
server_key_iv: "eb08f3a006fc10604f5766105c589e0b"
server_key_version: "encryptionkey@0"
I20260929 15:45:18.636950 8090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:18.654731 8090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:18.660604 8090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:18.660945 8090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:18.664834 8090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:18.665001 8090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:18.665215 8090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:18.665314 8090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:45:18 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:18.694780 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:18.665007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58585 (local address 127.0.97.62:38809)
0929 15:45:18.665848 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:18.665856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:18.667048 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 15:45:18.669620 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:18.669631 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:18.669765 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:18.669914 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:18.673268 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.674612 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.678232 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.679009 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.679215 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:18.686767 (+ 7552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:18.686774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:18.686777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:18.686819 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:18.690292 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:18.691304 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:18.691316 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:18.691320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:18.691503 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:18.692079 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:18.692085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:18.692088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:18.692471 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:18.692669 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:18.694135 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0929 15:45:18.694339 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":431,"threads_started":1}
I20260929 15:45:18.697294 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:18.665726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.1:58585)
0929 15:45:18.666800 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:18.666864 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:18.669162 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:18.670216 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:18.670226 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:18.671169 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:18.672836 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:18.672891 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.674927 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.674932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:18.677929 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:18.677971 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.679355 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.679370 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:18.679783 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:18.680758 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:18.680798 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:18.686389 (+ 5591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:18.690499 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:18.690511 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:18.690535 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:18.691144 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:18.691625 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:18.691630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:18.691634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:18.691949 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:18.692626 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:18.692635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:18.693389 (+ 754us) client_negotiation.cc:769] Sending connection context
0929 15:45:18.694020 (+ 631us) client_negotiation.cc:240] Negotiation successful
0929 15:45:18.694845 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":442,"threads_started":1}
I20260929 15:45:18.724666 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:18.706733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.1:34853)
0929 15:45:18.707936 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:18.707958 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:18.708212 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:18.708602 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:18.708606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:18.709140 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:18.709465 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:18.709475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.710522 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.710528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:18.711598 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:18.711612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.712599 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.712610 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:18.712906 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:18.714090 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:18.714134 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:18.716159 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:18.720632 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:18.720646 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:18.720656 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:18.721449 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:18.722172 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:18.722180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:18.722186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:18.722469 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:18.723204 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:18.723210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:18.723435 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 15:45:18.723609 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 15:45:18.723990 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":577,"threads_started":1}
I20260929 15:45:18.724866 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:18.706726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34853 (local address 127.0.97.62:38809)
0929 15:45:18.706948 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:18.706957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:18.708080 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 15:45:18.708374 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:18.708380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:18.708459 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:18.708572 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:18.709646 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.710368 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.711753 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:18.712306 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:18.712447 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:18.716616 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:18.716624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:18.716629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:18.716681 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:18.720322 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:18.721634 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:18.721639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:18.721642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:18.721814 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:18.722633 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:18.722639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:18.722643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:18.723067 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:18.723241 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:18.723850 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 15:45:18.724346 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:45:18.737063 8090 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38501
I20260929 15:45:18.737115 8225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38501 every 8 connection(s)
I20260929 15:45:18.738672 8090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:18.741884 8226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38809
I20260929 15:45:18.742318 8226 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:18.743309 8226 heartbeater.cc:511] Master 127.0.97.62:38809 requested a full tablet report, sending...
I20260929 15:45:18.745010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 8090
I20260929 15:45:18.745266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:18.745893 388 external_mini_cluster.cc:1428] Setting key 5a140d5c96e9bae0a8ddd20a5f02bb01
I20260929 15:45:18.746294 8019 ts_manager.cc:194] Registered new tserver with Master: 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501)
I20260929 15:45:18.748638 8019 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58585
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 15:45:18.803926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:19.206940 8230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:19.207362 8230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:19.207541 8230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:19.215610 8230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:19.215898 8230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:19.216068 8230 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:45:19.216259 8230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:19.231022 8230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:19.232684 8230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:19.235051 8230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:19.252081 8238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:19.252005 8236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:19.252125 8235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:19.253888 8230 server_base.cc:1034] running on GCE node
I20260929 15:45:19.254769 8230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:19.256244 8230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:19.257706 8230 hybrid_clock.cc:648] HybridClock initialized: now 1790696719257573 us; error 83 us; skew 500 ppm
I20260929 15:45:19.258184 8230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:19 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696719, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:19.262418 8230 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:45:19.264839 8230 webserver.cc:460] Webserver started at http://127.0.97.2:41835/ using document root <none> and password file <none>
I20260929 15:45:19.265759 8230 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:19.265949 8230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:19.266306 8230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:19.269722 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "b134051cd2a74e9c9c314942d2b2db13"
format_stamp: "Formatted at 2026-09-29 15:45:19 on dist-test-slave-7491"
server_key: "54d0ae102b89b1654782557f3a52d280"
server_key_iv: "82b2c04d7bd8d601bbb826d7dd6829ef"
server_key_version: "encryptionkey@0"
I20260929 15:45:19.270644 8230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b134051cd2a74e9c9c314942d2b2db13"
format_stamp: "Formatted at 2026-09-29 15:45:19 on dist-test-slave-7491"
server_key: "54d0ae102b89b1654782557f3a52d280"
server_key_iv: "82b2c04d7bd8d601bbb826d7dd6829ef"
server_key_version: "encryptionkey@0"
I20260929 15:45:19.278309 8230 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260929 15:45:19.283826 8245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:19.286266 8230 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 15:45:19.286688 8230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "b134051cd2a74e9c9c314942d2b2db13"
format_stamp: "Formatted at 2026-09-29 15:45:19 on dist-test-slave-7491"
server_key: "54d0ae102b89b1654782557f3a52d280"
server_key_iv: "82b2c04d7bd8d601bbb826d7dd6829ef"
server_key_version: "encryptionkey@0"
I20260929 15:45:19.287032 8230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:19.307267 8230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:19.313203 8230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:19.313575 8230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:19.316701 8230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:19.316820 8230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:19.316949 8230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:19.317041 8230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:45:19 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696719, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:19.347600 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:19.317354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39553 (local address 127.0.97.62:38809)
0929 15:45:19.317979 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:19.317984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:19.319692 (+ 1708us) server_negotiation.cc:407] Connection header received
0929 15:45:19.322537 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:19.322544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:19.322677 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:19.322994 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:19.327071 (+ 4077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.328232 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.331664 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.332513 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.332740 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:19.339082 (+ 6342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:19.339093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:19.339100 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:19.339156 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:19.343255 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:19.344639 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:19.344645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:19.344649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:19.344819 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:19.345573 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:19.345578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:19.345582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:19.345888 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:19.346060 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:19.346924 (+ 864us) server_negotiation.cc:299] Negotiation successful
0929 15:45:19.347193 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":413,"threads_started":1}
I20260929 15:45:19.349030 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:19.318194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.2:39553)
0929 15:45:19.319390 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:19.319459 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:19.322073 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:19.323730 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:19.323752 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:19.325127 (+ 1375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:19.326710 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:19.326758 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.328493 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.328503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:19.331430 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:19.331452 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.333369 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.333388 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:19.333691 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:19.334548 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:19.334575 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:19.338655 (+ 4080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:19.343631 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:19.343643 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:19.343666 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:19.344444 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:19.344968 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:19.344974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:19.344978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:19.345395 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:19.346057 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:19.346068 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:19.346506 (+ 438us) client_negotiation.cc:769] Sending connection context
0929 15:45:19.346812 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 15:45:19.347402 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":449,"threads_started":1}
I20260929 15:45:19.377247 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:19.358150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57955 (local address 127.0.97.62:38809)
0929 15:45:19.358452 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:19.358461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:19.359132 (+ 671us) server_negotiation.cc:407] Connection header received
0929 15:45:19.359456 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:19.359462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:19.359536 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:19.359599 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:19.361655 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.362820 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.365237 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.365746 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.365859 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:19.369454 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:19.369460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:19.369464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:19.369502 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:19.373281 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:19.374792 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:19.374796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:19.374799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:19.374935 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:19.375371 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:19.375376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:19.375378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:19.375739 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:19.375908 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:19.376628 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 15:45:19.376885 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:45:19.377285 8230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38941
I20260929 15:45:19.377324 8365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38941 every 8 connection(s)
I20260929 15:45:19.377446 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:19.357999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.2:57955)
0929 15:45:19.358975 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:19.358999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:19.359241 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:19.359886 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:19.359893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:19.360872 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:19.361423 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:19.361451 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.363014 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.363022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:19.364928 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:19.364945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.365995 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.366004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:19.366214 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:19.367576 (+ 1362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:19.367615 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:19.369064 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:19.373603 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:19.373615 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:19.373620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:19.374573 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:19.375076 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:19.375081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:19.375089 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:19.375231 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:19.375879 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:19.375885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:19.376295 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 15:45:19.376495 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 15:45:19.376916 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":673,"threads_started":1}
I20260929 15:45:19.378840 8230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:19.381618 8366 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38809
I20260929 15:45:19.381935 8366 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:19.382752 8366 heartbeater.cc:511] Master 127.0.97.62:38809 requested a full tablet report, sending...
I20260929 15:45:19.384667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 8230
I20260929 15:45:19.384708 8019 ts_manager.cc:194] Registered new tserver with Master: b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941)
I20260929 15:45:19.384887 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:19.385432 388 external_mini_cluster.cc:1428] Setting key 7efa843a01a39b4f6da87f551078f8aa
I20260929 15:45:19.385886 8019 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39553
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 15:45:19.441236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:19.753182 8226 heartbeater.cc:503] Master 127.0.97.62:38809 was elected leader, sending a full tablet report...
W20260929 15:45:19.862120 8370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:19.862447 8370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:19.862776 8370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:19.871459 8370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:19.871616 8370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:19.871719 8370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:45:19.871824 8370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:19.882689 8370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:19.884267 8370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:19.885730 8370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:19.901877 8375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:19.902071 8378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:19.901980 8376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:19.903599 8370 server_base.cc:1034] running on GCE node
I20260929 15:45:19.904558 8370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:19.905694 8370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:19.907147 8370 hybrid_clock.cc:648] HybridClock initialized: now 1790696719906967 us; error 51 us; skew 500 ppm
I20260929 15:45:19.907685 8370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:19 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696719, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:19.911583 8370 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:45:19.913427 8370 webserver.cc:460] Webserver started at http://127.0.97.3:45983/ using document root <none> and password file <none>
I20260929 15:45:19.914263 8370 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:19.914409 8370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:19.914757 8370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:19.918260 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "3d3693251a3b4763bf9c941eb48454e2"
format_stamp: "Formatted at 2026-09-29 15:45:19 on dist-test-slave-7491"
server_key: "22635d8ebcfcd1c660d79fe1bf408dfe"
server_key_iv: "1752f44ff44d15a8f8fe11d65accbd8e"
server_key_version: "encryptionkey@0"
I20260929 15:45:19.919193 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3d3693251a3b4763bf9c941eb48454e2"
format_stamp: "Formatted at 2026-09-29 15:45:19 on dist-test-slave-7491"
server_key: "22635d8ebcfcd1c660d79fe1bf408dfe"
server_key_iv: "1752f44ff44d15a8f8fe11d65accbd8e"
server_key_version: "encryptionkey@0"
I20260929 15:45:19.925484 8370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:45:19.929956 8385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:19.932019 8370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:45:19.932343 8370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "3d3693251a3b4763bf9c941eb48454e2"
format_stamp: "Formatted at 2026-09-29 15:45:19 on dist-test-slave-7491"
server_key: "22635d8ebcfcd1c660d79fe1bf408dfe"
server_key_iv: "1752f44ff44d15a8f8fe11d65accbd8e"
server_key_version: "encryptionkey@0"
I20260929 15:45:19.932650 8370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:19.963932 8370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:19.969894 8370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:19.970259 8370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:19.974436 8370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:19.974583 8370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:19.974818 8370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:19.974947 8370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:45:19 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696719, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:20.011039 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:19.975178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51327 (local address 127.0.97.62:38809)
0929 15:45:19.981379 (+ 6201us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:19.981388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:19.982532 (+ 1144us) server_negotiation.cc:407] Connection header received
0929 15:45:19.984976 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:19.984983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:19.985104 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:19.985237 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:19.987988 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.989194 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.992344 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.993043 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.993216 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.000101 (+ 6885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:20.000113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:20.000121 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:20.000226 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:20.005120 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.007063 (+ 1943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.007071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.007075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.007308 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.008273 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.008283 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.008291 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.008921 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:20.009126 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.010322 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.010623 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6003,"thread_start_us":5868,"threads_started":1}
I20260929 15:45:20.012362 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:19.976227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.3:51327)
0929 15:45:19.981997 (+ 5770us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:19.982063 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:19.984603 (+ 2540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:19.985582 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:19.985601 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:19.986567 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:19.987716 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:19.987759 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.989418 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.989429 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:19.992074 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:19.992115 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:19.993344 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:19.993353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:19.993720 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:19.994798 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:19.994838 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:19.999549 (+ 4711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:20.005378 (+ 5829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:20.005394 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:20.005447 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:20.006751 (+ 1304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:20.007516 (+ 765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:20.007526 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:20.007533 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:20.007993 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:20.009122 (+ 1129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:20.009134 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:20.009726 (+ 592us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.010167 (+ 441us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.010802 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5375,"thread_start_us":5004,"threads_started":1}
I20260929 15:45:20.043174 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.021275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33877 (local address 127.0.97.62:38809)
0929 15:45:20.021542 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.021551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.026036 (+ 4485us) server_negotiation.cc:407] Connection header received
0929 15:45:20.026283 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.026292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.026402 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.026521 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:20.028635 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.029726 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.031205 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.031634 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.031782 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.034838 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:20.034844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:20.034848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:20.034886 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:20.038842 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.039952 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.039956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.039959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.040099 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.040765 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.040772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.040776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.041620 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:20.041827 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.042672 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.042908 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 15:45:20.043678 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.021492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.3:33877)
0929 15:45:20.025747 (+ 4255us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.025833 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.026107 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.026623 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.026632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.027634 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:20.028392 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.028416 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.029914 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.029919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.031021 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.031037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.031859 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.031867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.032042 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.033093 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:20.033111 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:20.034536 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:20.039053 (+ 4517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:20.039059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:20.039063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:20.039740 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:20.040321 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:20.040330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:20.040338 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:20.040553 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:20.041837 (+ 1284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:20.041846 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:20.042174 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.042408 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.042950 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4046,"thread_start_us":3844,"threads_started":1}
I20260929 15:45:20.051458 8370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36333
I20260929 15:45:20.051509 8505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36333 every 8 connection(s)
I20260929 15:45:20.052925 8370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:20.055198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 8370
I20260929 15:45:20.055369 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:20.055876 388 external_mini_cluster.cc:1428] Setting key 084977a496d6fbec4afdb5cb956aa7d4
I20260929 15:45:20.055845 8506 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38809
I20260929 15:45:20.056394 8506 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:20.057587 8506 heartbeater.cc:511] Master 127.0.97.62:38809 requested a full tablet report, sending...
I20260929 15:45:20.059309 8018 ts_manager.cc:194] Registered new tserver with Master: 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:20.060477 8018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51327
I20260929 15:45:20.074183 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:45:20 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:45:20 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:20 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:45:20.135074 8394 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:20.122006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.0.97.62:38809)
0929 15:45:20.122390 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.122400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.122830 (+ 430us) server_negotiation.cc:407] Connection header received
0929 15:45:20.122998 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.123002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.123090 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.123200 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:20.125045 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.125806 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.127127 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.127536 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.127698 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.134527 (+ 6829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51250: BlockingRecv error: recv got EOF from 127.0.0.1:51250 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 29 15:45:20 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:20.158396 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.140950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.0.97.62:38809)
0929 15:45:20.141209 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.141218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.141713 (+ 495us) server_negotiation.cc:407] Connection header received
0929 15:45:20.142115 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.142123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.142244 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.142417 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:20.144033 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.145068 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.146620 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.147082 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.147216 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.151547 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:20.151554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:20.151558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:20.151603 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:20.154631 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.155560 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.155567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.155580 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.155825 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.156652 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.156656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.156659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.157037 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:20.157318 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.157895 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.158150 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:45:20.164490 8018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:20.173503 8018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:20.223722 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.192509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38501 (local address 127.0.0.1:54336)
0929 15:45:20.197447 (+ 4938us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.197479 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.197709 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.201580 (+ 3871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.201591 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.201643 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.202379 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.202400 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.207579 (+ 5179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.207591 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.215267 (+ 7676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.215314 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.222214 (+ 6900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.222223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.222558 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.222644 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.222868 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.223268 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4784,"mutex_wait_us":102,"thread_start_us":1529,"threads_started":1}
I20260929 15:45:20.223762 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.192982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.0.97.1:38501)
0929 15:45:20.196887 (+ 3905us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.196903 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.197908 (+ 1005us) server_negotiation.cc:407] Connection header received
0929 15:45:20.198076 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.198229 (+ 153us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.200939 (+ 2710us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.201213 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.202673 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.207265 (+ 4592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.215776 (+ 8511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.221828 (+ 6052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.222068 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.222463 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.223109 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.223387 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3643,"thread_start_us":3454,"threads_started":1}
I20260929 15:45:20.229086 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.192404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.0.97.2:38941)
0929 15:45:20.221056 (+ 28652us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.221065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.221100 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:45:20.221230 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.221242 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.221544 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.221800 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.222604 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.224151 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.226709 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.227899 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.228080 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.228350 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.228595 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.228780 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28430,"thread_start_us":28297,"threads_started":1}
I20260929 15:45:20.229128 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.192039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38941 (local address 127.0.0.1:50940)
0929 15:45:20.193937 (+ 1898us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.193992 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.194239 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.221793 (+ 27554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.221798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.221827 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.222330 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.222343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.224333 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.224340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.226479 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.226496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.228151 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.228192 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.228369 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.228408 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.228547 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.228703 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":1,"thread_start_us":5041,"threads_started":1}
I20260929 15:45:20.229046 8156 tablet_service.cc:1467] Processing CreateTablet for tablet d2d7fcda6fd345e89bda8301904cf5b8 (DEFAULT_TABLE table=test-table [id=91174f3c7f9342deac349012b4b46e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:20.231205 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.193019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.0.1:46022)
0929 15:45:20.220896 (+ 27877us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.220919 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.221109 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.222514 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.222521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.222558 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.223300 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.223318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.225751 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.225757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.228005 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.228022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.230325 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.230333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.230533 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.230567 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.230698 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.230851 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27709,"thread_start_us":27587,"threads_started":1}
I20260929 15:45:20.231220 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d7fcda6fd345e89bda8301904cf5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.231288 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.193735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.97.3:36333)
0929 15:45:20.220426 (+ 26691us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.220438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.221188 (+ 750us) server_negotiation.cc:407] Connection header received
0929 15:45:20.221394 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.221403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.221760 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.222070 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.223651 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.225562 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.228290 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.230024 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.230183 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.230436 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.230756 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.230933 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26477,"thread_start_us":26239,"threads_started":1}
I20260929 15:45:20.233247 8296 tablet_service.cc:1467] Processing CreateTablet for tablet d2d7fcda6fd345e89bda8301904cf5b8 (DEFAULT_TABLE table=test-table [id=91174f3c7f9342deac349012b4b46e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:20.234957 8296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d7fcda6fd345e89bda8301904cf5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.235606 8436 tablet_service.cc:1467] Processing CreateTablet for tablet d2d7fcda6fd345e89bda8301904cf5b8 (DEFAULT_TABLE table=test-table [id=91174f3c7f9342deac349012b4b46e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:20.237708 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d7fcda6fd345e89bda8301904cf5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.248570 8530 tablet_bootstrap.cc:492] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Bootstrap starting.
I20260929 15:45:20.253242 8530 tablet_bootstrap.cc:654] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.254768 8530 log.cc:824] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:20.255744 8532 tablet_bootstrap.cc:492] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Bootstrap starting.
I20260929 15:45:20.257017 8531 tablet_bootstrap.cc:492] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Bootstrap starting.
I20260929 15:45:20.262151 8531 tablet_bootstrap.cc:654] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.263365 8531 log.cc:824] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:20.264415 8530 tablet_bootstrap.cc:492] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: No bootstrap required, opened a new log
I20260929 15:45:20.264782 8530 ts_tablet_manager.cc:1397] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20260929 15:45:20.264894 8532 tablet_bootstrap.cc:654] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.266340 8532 log.cc:824] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:20.266774 8531 tablet_bootstrap.cc:492] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: No bootstrap required, opened a new log
I20260929 15:45:20.267148 8531 ts_tablet_manager.cc:1397] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260929 15:45:20.270229 8532 tablet_bootstrap.cc:492] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: No bootstrap required, opened a new log
I20260929 15:45:20.270592 8532 ts_tablet_manager.cc:1397] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260929 15:45:20.276283 8531 raft_consensus.cc:348] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.276319 8530 raft_consensus.cc:348] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.276814 8530 raft_consensus.cc:374] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.276871 8531 raft_consensus.cc:374] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.276971 8530 raft_consensus.cc:729] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.276997 8531 raft_consensus.cc:729] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.277773 8530 consensus_queue.cc:260] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.277801 8531 consensus_queue.cc:260] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.280679 8530 ts_tablet_manager.cc:1428] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260929 15:45:20.281836 8366 heartbeater.cc:503] Master 127.0.97.62:38809 was elected leader, sending a full tablet report...
I20260929 15:45:20.282724 8531 ts_tablet_manager.cc:1428] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260929 15:45:20.283608 8532 raft_consensus.cc:348] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.284025 8532 raft_consensus.cc:374] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.284143 8532 raft_consensus.cc:729] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3693251a3b4763bf9c941eb48454e2, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.284786 8532 consensus_queue.cc:260] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.285976 8506 heartbeater.cc:503] Master 127.0.97.62:38809 was elected leader, sending a full tablet report...
I20260929 15:45:20.287483 8532 ts_tablet_manager.cc:1428] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260929 15:45:20.305012 8507 tablet.cc:2369] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:45:20.381524 8367 tablet.cc:2369] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:20.393828 8537 raft_consensus.cc:482] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:20.394285 8537 raft_consensus.cc:504] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.396863 8537 leader_election.cc:288] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:20.406129 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.397393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.97.1:52425)
0929 15:45:20.399000 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.399042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.399331 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.399791 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.399795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.399824 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.400379 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.400391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.401643 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.401654 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.403693 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.403715 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.405203 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.405211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.405405 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.405455 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.405559 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.405750 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":1171,"threads_started":1}
I20260929 15:45:20.406425 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.397990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52425 (local address 127.0.97.3:36333)
0929 15:45:20.398245 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.398254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.399178 (+ 924us) server_negotiation.cc:407] Connection header received
0929 15:45:20.399562 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.399573 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.399639 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.399772 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.400575 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.401479 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.403910 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.404932 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.405046 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.405170 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.405889 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.406096 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:45:20.406852 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.397474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38941 (local address 127.0.97.1:51629)
0929 15:45:20.399470 (+ 1996us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.399493 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.399755 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.400252 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.400257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.400286 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.400739 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.400791 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.402415 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.402420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.404150 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.404165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.405806 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.405815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.406041 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.406092 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.406234 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.406408 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1811,"mutex_wait_us":65,"thread_start_us":1575,"threads_started":1}
I20260929 15:45:20.406847 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.397472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51629 (local address 127.0.97.2:38941)
0929 15:45:20.397745 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.397755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.399669 (+ 1914us) server_negotiation.cc:407] Connection header received
0929 15:45:20.399927 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.399932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.400003 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.400142 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.400968 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.402201 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.404389 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.405390 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.405560 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.405663 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.406460 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.406622 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:45:20.407996 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7fcda6fd345e89bda8301904cf5b8" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2" is_pre_election: true
I20260929 15:45:20.408331 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7fcda6fd345e89bda8301904cf5b8" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13" is_pre_election: true
I20260929 15:45:20.408730 8460 raft_consensus.cc:2454] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 0.
I20260929 15:45:20.408855 8317 raft_consensus.cc:2454] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 0.
I20260929 15:45:20.409565 8107 leader_election.cc:302] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, 3d3693251a3b4763bf9c941eb48454e2; no voters:
I20260929 15:45:20.410002 8537 raft_consensus.cc:2781] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:20.410143 8537 raft_consensus.cc:482] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:20.410310 8537 raft_consensus.cc:3037] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:20.413182 8537 raft_consensus.cc:504] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.414067 8537 leader_election.cc:288] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 election: Requested vote from peers b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:20.414942 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7fcda6fd345e89bda8301904cf5b8" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13"
I20260929 15:45:20.414942 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d7fcda6fd345e89bda8301904cf5b8" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2"
I20260929 15:45:20.415225 8460 raft_consensus.cc:3037] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:20.415223 8317 raft_consensus.cc:3037] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:20.419271 8460 raft_consensus.cc:2454] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 1.
I20260929 15:45:20.419271 8317 raft_consensus.cc:2454] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 1.
I20260929 15:45:20.419911 8107 leader_election.cc:302] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, 3d3693251a3b4763bf9c941eb48454e2; no voters:
I20260929 15:45:20.420356 8537 raft_consensus.cc:2781] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:20.420965 8537 raft_consensus.cc:686] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Running, Role: LEADER
I20260929 15:45:20.421610 8537 consensus_queue.cc:237] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.428015 8017 catalog_manager.cc:5495] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 15:45:20.455088 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.449071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.0.97.1:38501)
0929 15:45:20.449328 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.449337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.449360 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:45:20.449654 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.449658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.449772 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.449953 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:20.450324 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.451039 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.452652 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.453164 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.453313 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.453750 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:20.454407 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:20.454560 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.454712 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.454800 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:45:20.465935 8317 raft_consensus.cc:1261] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Refusing update from remote peer 1d23bbf1b5774e52ab4ad0b33b0541bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:20.465935 8460 raft_consensus.cc:1261] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Refusing update from remote peer 1d23bbf1b5774e52ab4ad0b33b0541bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:20.467218 8541 consensus_queue.cc:1035] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:20.467685 8537 consensus_queue.cc:1035] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:20.480262 8546 mvcc.cc:205] Tried to move back new op lower bound from 7334693767014588416 to 7334693766852464640. Current Snapshot: MvccSnapshot[applied={T|T < 7334693767014588416}]
I20260929 15:45:20.508088 8017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:20.508497 8017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:20.513106 8017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Sending DeleteTablet for 3 replicas of tablet d2d7fcda6fd345e89bda8301904cf5b8
I20260929 15:45:20.514855 8296 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d7fcda6fd345e89bda8301904cf5b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50940
I20260929 15:45:20.514904 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d7fcda6fd345e89bda8301904cf5b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260929 15:45:20.514914 8156 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d7fcda6fd345e89bda8301904cf5b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54336
I20260929 15:45:20.519649 8017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:20.520514 8017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:20.527901 8554 tablet_replica.cc:330] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: stopping tablet replica
I20260929 15:45:20.529484 8554 raft_consensus.cc:2229] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:20.530053 8554 raft_consensus.cc:2258] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:20.530112 8556 tablet_replica.cc:330] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: stopping tablet replica
I20260929 15:45:20.530566 8556 raft_consensus.cc:2229] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:20.531260 8556 raft_consensus.cc:2258] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:20.531951 8555 tablet_replica.cc:330] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: stopping tablet replica
I20260929 15:45:20.532445 8555 raft_consensus.cc:2229] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:20.532934 8555 raft_consensus.cc:2258] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:20.533733 8554 ts_tablet_manager.cc:1905] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:20.535105 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 (DEFAULT_TABLE table=test-table [id=f8d3dac089a94f7aaf6774821f0fc458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:20.535079 8296 tablet_service.cc:1467] Processing CreateTablet for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 (DEFAULT_TABLE table=test-table [id=f8d3dac089a94f7aaf6774821f0fc458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:20.535571 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.535571 8296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.536185 8555 ts_tablet_manager.cc:1905] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:20.541767 8156 tablet_service.cc:1467] Processing CreateTablet for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 (DEFAULT_TABLE table=test-table [id=f8d3dac089a94f7aaf6774821f0fc458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:20.542443 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.543491 8556 ts_tablet_manager.cc:1905] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:20.545032 8555 ts_tablet_manager.cc:1918] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:20.545293 8555 log.cc:1196] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/d2d7fcda6fd345e89bda8301904cf5b8
I20260929 15:45:20.546181 8555 ts_tablet_manager.cc:1939] T d2d7fcda6fd345e89bda8301904cf5b8 P 3d3693251a3b4763bf9c941eb48454e2: Deleting consensus metadata
I20260929 15:45:20.547896 8531 tablet_bootstrap.cc:492] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Bootstrap starting.
I20260929 15:45:20.549809 8531 tablet_bootstrap.cc:654] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.554243 8530 tablet_bootstrap.cc:492] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Bootstrap starting.
I20260929 15:45:20.556677 8530 tablet_bootstrap.cc:654] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.558840 8004 catalog_manager.cc:4841] TS 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333): tablet d2d7fcda6fd345e89bda8301904cf5b8 (table test-table [id=91174f3c7f9342deac349012b4b46e78]) successfully deleted
I20260929 15:45:20.562923 8530 tablet_bootstrap.cc:492] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: No bootstrap required, opened a new log
I20260929 15:45:20.563169 8530 ts_tablet_manager.cc:1397] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 15:45:20.564446 8530 raft_consensus.cc:348] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.564733 8530 raft_consensus.cc:374] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.564795 8531 tablet_bootstrap.cc:492] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: No bootstrap required, opened a new log
I20260929 15:45:20.564846 8530 raft_consensus.cc:729] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.564997 8531 ts_tablet_manager.cc:1397] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260929 15:45:20.565212 8530 consensus_queue.cc:260] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.565680 8554 ts_tablet_manager.cc:1918] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:20.566025 8554 log.cc:1196] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/d2d7fcda6fd345e89bda8301904cf5b8
I20260929 15:45:20.566264 8531 raft_consensus.cc:348] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.566685 8531 raft_consensus.cc:374] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.566825 8531 raft_consensus.cc:729] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.566933 8554 ts_tablet_manager.cc:1939] T d2d7fcda6fd345e89bda8301904cf5b8 P b134051cd2a74e9c9c314942d2b2db13: Deleting consensus metadata
I20260929 15:45:20.568495 8532 tablet_bootstrap.cc:492] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Bootstrap starting.
I20260929 15:45:20.551483 8556 ts_tablet_manager.cc:1918] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:20.569721 8556 log.cc:1196] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/d2d7fcda6fd345e89bda8301904cf5b8
I20260929 15:45:20.569864 8530 ts_tablet_manager.cc:1428] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 15:45:20.570384 8556 ts_tablet_manager.cc:1939] T d2d7fcda6fd345e89bda8301904cf5b8 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting consensus metadata
I20260929 15:45:20.570595 8532 tablet_bootstrap.cc:654] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.573390 8005 catalog_manager.cc:4841] TS b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941): tablet d2d7fcda6fd345e89bda8301904cf5b8 (table test-table [id=91174f3c7f9342deac349012b4b46e78]) successfully deleted
I20260929 15:45:20.573361 8531 consensus_queue.cc:260] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.575039 8531 ts_tablet_manager.cc:1428] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20260929 15:45:20.575826 8003 catalog_manager.cc:4841] TS 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501): tablet d2d7fcda6fd345e89bda8301904cf5b8 (table test-table [id=91174f3c7f9342deac349012b4b46e78]) successfully deleted
I20260929 15:45:20.577721 8532 tablet_bootstrap.cc:492] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: No bootstrap required, opened a new log
I20260929 15:45:20.578008 8532 ts_tablet_manager.cc:1397] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 15:45:20.579067 8532 raft_consensus.cc:348] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.579345 8532 raft_consensus.cc:374] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.579448 8532 raft_consensus.cc:729] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3693251a3b4763bf9c941eb48454e2, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.579793 8532 consensus_queue.cc:260] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.580709 8532 ts_tablet_manager.cc:1428] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260929 15:45:20.742296 8536 raft_consensus.cc:482] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:20.742569 8536 raft_consensus.cc:504] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
W20260929 15:45:20.743234 8227 tablet.cc:2369] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:20.744868 8536 leader_election.cc:288] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:20.762025 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.745327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35347 (local address 127.0.97.1:38501)
0929 15:45:20.745519 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.745525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.753060 (+ 7535us) server_negotiation.cc:407] Connection header received
0929 15:45:20.753308 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.753315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.753398 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.753492 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.754425 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.755680 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.758787 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.760133 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.760326 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.760451 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.761123 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.761273 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:45:20.762087 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.745827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43901 (local address 127.0.97.3:36333)
0929 15:45:20.746499 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.746507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.753320 (+ 6813us) server_negotiation.cc:407] Connection header received
0929 15:45:20.753539 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.753544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.753626 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.753716 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:20.754589 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.755771 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.759269 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.760576 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.760682 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.760784 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.761391 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.761715 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525}
I20260929 15:45:20.762135 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.744948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38501 (local address 127.0.97.2:35347)
0929 15:45:20.752860 (+ 7912us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.752892 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.753124 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.753666 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.753672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.753700 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.754122 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.754134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.755897 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.755902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.758561 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.758582 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.760683 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.760690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.760883 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.760922 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.761495 (+ 573us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.761675 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7693,"mutex_wait_us":9,"thread_start_us":7921,"threads_started":1}
I20260929 15:45:20.763171 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.745358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.97.2:43901)
0929 15:45:20.753215 (+ 7857us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.753234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.753402 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.753968 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.753973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.753997 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:20.754413 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.754424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.755971 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.755976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.758623 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.758640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.760990 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.760996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.761209 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.761240 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.761969 (+ 729us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.762143 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7721,"mutex_wait_us":227,"thread_start_us":7062,"threads_started":1}
I20260929 15:45:20.764110 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4ec4c3c0344fd8b2ec9024990e6d19" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" is_pre_election: true
I20260929 15:45:20.764753 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4ec4c3c0344fd8b2ec9024990e6d19" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2" is_pre_election: true
I20260929 15:45:20.764787 8179 raft_consensus.cc:2454] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 0.
I20260929 15:45:20.765094 8460 raft_consensus.cc:2454] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 0.
I20260929 15:45:20.765704 8246 leader_election.cc:302] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, b134051cd2a74e9c9c314942d2b2db13; no voters:
I20260929 15:45:20.766274 8536 raft_consensus.cc:2781] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:20.766414 8536 raft_consensus.cc:482] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:20.766518 8536 raft_consensus.cc:3037] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:20.770831 8536 raft_consensus.cc:504] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.772473 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4ec4c3c0344fd8b2ec9024990e6d19" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2"
I20260929 15:45:20.772730 8460 raft_consensus.cc:3037] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:20.773067 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4ec4c3c0344fd8b2ec9024990e6d19" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd"
I20260929 15:45:20.773343 8179 raft_consensus.cc:3037] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:20.775911 8460 raft_consensus.cc:2454] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 1.
I20260929 15:45:20.776393 8536 leader_election.cc:288] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 election: Requested vote from peers 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:20.776583 8247 leader_election.cc:302] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3693251a3b4763bf9c941eb48454e2, b134051cd2a74e9c9c314942d2b2db13; no voters:
I20260929 15:45:20.776897 8536 raft_consensus.cc:2781] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:20.777094 8179 raft_consensus.cc:2454] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 1.
I20260929 15:45:20.777590 8536 raft_consensus.cc:686] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 1 LEADER]: Becoming Leader. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Running, Role: LEADER
I20260929 15:45:20.778056 8536 consensus_queue.cc:237] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.782716 8017 catalog_manager.cc:5495] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
Sep 29 15:45:20 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:20.852043 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.830929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.0.1:51264)
0929 15:45:20.831884 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:20.831902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:20.832040 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:20.832620 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:20.832624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:20.833388 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:20.833650 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.833661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.835262 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.835267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.837204 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:20.837234 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.838536 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.838543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:20.838731 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.839702 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:20.839725 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:20.842496 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:20.848021 (+ 5525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:20.848042 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:20.848050 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:20.848877 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:20.849652 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:20.849658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:20.849662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:20.849896 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:20.850777 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:20.850787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:20.850988 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 15:45:20.851189 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 15:45:20.851711 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":641,"threads_started":1}
I20260929 15:45:20.852658 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:20.830977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.0.97.62:38809)
0929 15:45:20.831662 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:20.831669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:20.832008 (+ 339us) server_negotiation.cc:407] Connection header received
0929 15:45:20.832191 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:20.832197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:20.832294 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:20.832441 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:20.833843 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.835035 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.837492 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:20.838255 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:20.838440 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:20.842868 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:20.842879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:20.842885 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:20.842946 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:20.847663 (+ 4717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.849097 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.849106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.849113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.849426 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:20.850076 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:20.850082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:20.850086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:20.850549 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:20.850756 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:20.851379 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 15:45:20.851964 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":243,"server-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260929 15:45:20.860503 8017 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:45:20.862061 8017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:20.871824 8156 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbfe4466484444ca673315efacbc124 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f2c48e294ec4876bca7b5b65daf1507]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:45:20.872023 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbfe4466484444ca673315efacbc124 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f2c48e294ec4876bca7b5b65daf1507]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:45:20.871999 8296 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbfe4466484444ca673315efacbc124 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f2c48e294ec4876bca7b5b65daf1507]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:45:20.872584 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbfe4466484444ca673315efacbc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.872614 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbfe4466484444ca673315efacbc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.872653 8296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbfe4466484444ca673315efacbc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:20.881672 8532 tablet_bootstrap.cc:492] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: Bootstrap starting.
I20260929 15:45:20.881672 8531 tablet_bootstrap.cc:492] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: Bootstrap starting.
I20260929 15:45:20.883670 8531 tablet_bootstrap.cc:654] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.883670 8532 tablet_bootstrap.cc:654] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.884423 8530 tablet_bootstrap.cc:492] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Bootstrap starting.
I20260929 15:45:20.886535 8530 tablet_bootstrap.cc:654] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:20.894330 8532 tablet_bootstrap.cc:492] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: No bootstrap required, opened a new log
I20260929 15:45:20.894590 8532 ts_tablet_manager.cc:1397] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260929 15:45:20.894786 8531 tablet_bootstrap.cc:492] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: No bootstrap required, opened a new log
I20260929 15:45:20.895061 8531 ts_tablet_manager.cc:1397] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260929 15:45:20.895563 8532 raft_consensus.cc:348] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.895853 8532 raft_consensus.cc:374] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.895941 8532 raft_consensus.cc:729] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3693251a3b4763bf9c941eb48454e2, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.896090 8530 tablet_bootstrap.cc:492] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: No bootstrap required, opened a new log
I20260929 15:45:20.896303 8530 ts_tablet_manager.cc:1397] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 15:45:20.896245 8532 consensus_queue.cc:260] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.896391 8531 raft_consensus.cc:348] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.896704 8531 raft_consensus.cc:374] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.896782 8531 raft_consensus.cc:729] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.896765 8538 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:20.897089 8538 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:20.897163 8530 raft_consensus.cc:348] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.897411 8530 raft_consensus.cc:374] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:20.897505 8530 raft_consensus.cc:729] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Initialized, Role: FOLLOWER
I20260929 15:45:20.897764 8530 consensus_queue.cc:260] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.898653 8532 ts_tablet_manager.cc:1428] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:20.898662 8537 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:20.898952 8537 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:20.899617 8530 ts_tablet_manager.cc:1428] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:45:20.898185 8531 consensus_queue.cc:260] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:20.901906 8536 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:20.902318 8536 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:20.904695 8531 ts_tablet_manager.cc:1428] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260929 15:45:21.061728 8536 raft_consensus.cc:482] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:21.062047 8536 raft_consensus.cc:504] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.063005 8536 leader_election.cc:288] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:21.063776 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbfe4466484444ca673315efacbc124" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" is_pre_election: true
I20260929 15:45:21.063864 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbfe4466484444ca673315efacbc124" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2" is_pre_election: true
I20260929 15:45:21.064122 8179 raft_consensus.cc:2454] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 0.
I20260929 15:45:21.064157 8460 raft_consensus.cc:2454] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 0.
I20260929 15:45:21.065001 8246 leader_election.cc:302] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, 3d3693251a3b4763bf9c941eb48454e2, b134051cd2a74e9c9c314942d2b2db13; no voters:
I20260929 15:45:21.065474 8536 raft_consensus.cc:2781] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:21.065595 8536 raft_consensus.cc:482] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:21.065698 8536 raft_consensus.cc:3037] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.068745 8536 raft_consensus.cc:504] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.069919 8536 leader_election.cc:288] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 election: Requested vote from peers 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:21.070443 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbfe4466484444ca673315efacbc124" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd"
I20260929 15:45:21.070447 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbfe4466484444ca673315efacbc124" candidate_uuid: "b134051cd2a74e9c9c314942d2b2db13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2"
I20260929 15:45:21.070748 8460 raft_consensus.cc:3037] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.070904 8179 raft_consensus.cc:3037] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.073232 8179 raft_consensus.cc:2454] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 1.
I20260929 15:45:21.074056 8246 leader_election.cc:302] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, b134051cd2a74e9c9c314942d2b2db13; no voters:
I20260929 15:45:21.074110 8460 raft_consensus.cc:2454] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b134051cd2a74e9c9c314942d2b2db13 in term 1.
I20260929 15:45:21.074597 8536 raft_consensus.cc:2781] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:21.074801 8536 raft_consensus.cc:686] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [term 1 LEADER]: Becoming Leader. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Running, Role: LEADER
I20260929 15:45:21.075098 8536 consensus_queue.cc:237] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.077139 8562 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: TxnStatusTablet state changed. Reason: New leader b134051cd2a74e9c9c314942d2b2db13. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.077498 8562 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:45:21.078186 8575 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:45:21.078474 8575 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:45:21.079293 8019 catalog_manager.cc:5495] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 reported cstate change: term changed from 0 to 1, leader changed from <none> to b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 15:45:21.109599 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.102248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38941 (local address 127.0.0.1:50956)
0929 15:45:21.102697 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:21.102744 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:21.103006 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:21.104132 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:21.104136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:21.104151 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:45:21.104442 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.104452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.105649 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.105655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.106673 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.106686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.107519 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.107525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.107803 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.107874 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.108957 (+ 1083us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.108980 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:45:21.109073 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:45:21.109276 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 15:45:21.109638 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.102257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.0.97.2:38941)
0929 15:45:21.103703 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.103707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.103734 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:45:21.103825 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.103828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.103918 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.104111 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:21.104654 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.105499 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.106851 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.107331 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.107489 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.108090 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.108831 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.109000 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.109112 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.109199 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":1124,"threads_started":1}
I20260929 15:45:21.115100 8460 raft_consensus.cc:1261] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Refusing update from remote peer b134051cd2a74e9c9c314942d2b2db13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.115482 8179 raft_consensus.cc:1261] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Refusing update from remote peer b134051cd2a74e9c9c314942d2b2db13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.115933 8562 consensus_queue.cc:1035] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.116366 8536 consensus_queue.cc:1035] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.133059 8538 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: TxnStatusTablet state changed. Reason: New leader b134051cd2a74e9c9c314942d2b2db13. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.133431 8538 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:21.133559 8537 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: TxnStatusTablet state changed. Reason: New leader b134051cd2a74e9c9c314942d2b2db13. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.133870 8537 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:21.137375 8562 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.137706 8562 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:45:21.137704 8580 mvcc.cc:205] Tried to move back new op lower bound from 7334693769679351808 to 7334693769526894592. Current Snapshot: MvccSnapshot[applied={T|T < 7334693769679351808}]
I20260929 15:45:21.137961 8562 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.138260 8562 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P b134051cd2a74e9c9c314942d2b2db13: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:45:21.144649 8538 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.145198 8538 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P 3d3693251a3b4763bf9c941eb48454e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:21.150331 8537 tablet_replica.cc:439] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b134051cd2a74e9c9c314942d2b2db13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 15:45:21.150645 8537 tablet_replica.cc:442] T 5cbfe4466484444ca673315efacbc124 P 1d23bbf1b5774e52ab4ad0b33b0541bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:45:21.157358 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.148325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.0.97.2:38941)
0929 15:45:21.148502 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.148508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.149728 (+ 1220us) server_negotiation.cc:407] Connection header received
0929 15:45:21.149845 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.149851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.149941 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.150045 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:21.150800 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.152036 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.154990 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.155642 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.155771 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.156439 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.156558 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.156648 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.156914 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.156993 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:45:21.180604 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.171571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34077 (local address 127.0.97.2:38941)
0929 15:45:21.171880 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.171888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.172945 (+ 1057us) server_negotiation.cc:407] Connection header received
0929 15:45:21.173134 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.173139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.173248 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.173350 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:21.174112 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.174988 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.177451 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.178354 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.178551 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.179361 (+ 810us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.179675 (+ 314us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.179787 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.180023 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.180270 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":92}
I20260929 15:45:21.181061 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.171571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38941 (local address 127.0.97.2:34077)
0929 15:45:21.172721 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:21.172753 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:21.172973 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:21.173403 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:21.173408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:21.173421 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:45:21.173858 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.173873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.175192 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.175199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.177205 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.177230 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.178693 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.178702 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.178929 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.179044 (+ 115us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.179829 (+ 785us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.179849 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 15:45:21.179951 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:45:21.180263 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":90,"thread_start_us":621,"threads_started":1}
I20260929 15:45:21.194880 8179 raft_consensus.cc:1261] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Refusing update from remote peer b134051cd2a74e9c9c314942d2b2db13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.195111 8460 raft_consensus.cc:1261] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Refusing update from remote peer b134051cd2a74e9c9c314942d2b2db13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.195659 8584 consensus_queue.cc:1035] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.196125 8536 consensus_queue.cc:1035] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.210798 8581 mvcc.cc:205] Tried to move back new op lower bound from 7334693770006732800 to 7334693768311472128. Current Snapshot: MvccSnapshot[applied={T|T < 7334693770006732800}]
I20260929 15:45:21.276990 8017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:21.277520 8017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:21.282032 8017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Sending DeleteTablet for 3 replicas of tablet 1a4ec4c3c0344fd8b2ec9024990e6d19
I20260929 15:45:21.283174 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260929 15:45:21.283401 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50940
I20260929 15:45:21.285569 8156 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54336
I20260929 15:45:21.300823 8598 tablet_replica.cc:330] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: stopping tablet replica
I20260929 15:45:21.301281 8598 raft_consensus.cc:2229] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:21.301769 8598 raft_consensus.cc:2258] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:21.303469 8598 ts_tablet_manager.cc:1905] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:21.304752 8597 tablet_replica.cc:330] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: stopping tablet replica
I20260929 15:45:21.304738 8596 tablet_replica.cc:330] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: stopping tablet replica
I20260929 15:45:21.304965 8597 raft_consensus.cc:2229] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:21.304965 8596 raft_consensus.cc:2229] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:21.305248 8596 raft_consensus.cc:2258] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:21.306032 8597 raft_consensus.cc:2258] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:21.307303 8597 ts_tablet_manager.cc:1905] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:21.307746 8596 ts_tablet_manager.cc:1905] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:21.312674 8598 ts_tablet_manager.cc:1918] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:45:21.312963 8598 log.cc:1196] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/1a4ec4c3c0344fd8b2ec9024990e6d19
I20260929 15:45:21.313552 8598 ts_tablet_manager.cc:1939] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting consensus metadata
I20260929 15:45:21.314968 8003 catalog_manager.cc:4841] TS 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501): tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 (table test-table [id=f8d3dac089a94f7aaf6774821f0fc458]) successfully deleted
I20260929 15:45:21.315816 8597 ts_tablet_manager.cc:1918] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:45:21.315801 8596 ts_tablet_manager.cc:1918] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:45:21.315964 8596 log.cc:1196] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/1a4ec4c3c0344fd8b2ec9024990e6d19
I20260929 15:45:21.316414 8597 log.cc:1196] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/1a4ec4c3c0344fd8b2ec9024990e6d19
I20260929 15:45:21.316915 8596 ts_tablet_manager.cc:1939] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P 3d3693251a3b4763bf9c941eb48454e2: Deleting consensus metadata
I20260929 15:45:21.317166 8597 ts_tablet_manager.cc:1939] T 1a4ec4c3c0344fd8b2ec9024990e6d19 P b134051cd2a74e9c9c314942d2b2db13: Deleting consensus metadata
I20260929 15:45:21.318507 8004 catalog_manager.cc:4841] TS 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333): tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 (table test-table [id=f8d3dac089a94f7aaf6774821f0fc458]) successfully deleted
I20260929 15:45:21.318696 8005 catalog_manager.cc:4841] TS b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941): tablet 1a4ec4c3c0344fd8b2ec9024990e6d19 (table test-table [id=f8d3dac089a94f7aaf6774821f0fc458]) successfully deleted
Sep 29 15:45:21 dist-test-slave-7491 krb5kdc[7974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:45:21 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:21 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:45:21.376881 8605 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:21.355656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.97.62:38809)
0929 15:45:21.356617 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.356625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.357052 (+ 427us) server_negotiation.cc:407] Connection header received
0929 15:45:21.357252 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.357258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.357345 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.357480 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:21.359296 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.360580 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.362258 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.362933 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.363123 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.376527 (+ 13404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51270: BlockingRecv error: recv got EOF from 127.0.0.1:51270 (error 108)
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":686,"threads_started":1}
Sep 29 15:45:21 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 15:45:21.397837 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.381291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.0.97.62:38809)
0929 15:45:21.381520 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.381529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.382086 (+ 557us) server_negotiation.cc:407] Connection header received
0929 15:45:21.382307 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.382318 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.382384 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.382497 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:21.383787 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.384757 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.386100 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.386529 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.386677 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.391124 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:21.391130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:21.391134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:21.391191 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:21.394635 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:21.395477 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:21.395483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:21.395487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:21.395639 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:21.396053 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:21.396059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:21.396062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:21.396449 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:21.396769 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.397094 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.397338 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:45:21.402345 8017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:21.403146 8017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:21.411410 8156 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 (DEFAULT_TABLE table=test-table [id=19864f9c7c0140669a66f6b65ad2b960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:21.411767 8295 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 (DEFAULT_TABLE table=test-table [id=19864f9c7c0140669a66f6b65ad2b960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:21.411909 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:21.412300 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:21.412147 8436 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 (DEFAULT_TABLE table=test-table [id=19864f9c7c0140669a66f6b65ad2b960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:21.412582 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:21.426852 8612 tablet_bootstrap.cc:492] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Bootstrap starting.
I20260929 15:45:21.428936 8612 tablet_bootstrap.cc:654] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:21.429466 8613 tablet_bootstrap.cc:492] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Bootstrap starting.
I20260929 15:45:21.432366 8613 tablet_bootstrap.cc:654] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:21.433503 8614 tablet_bootstrap.cc:492] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Bootstrap starting.
I20260929 15:45:21.436198 8614 tablet_bootstrap.cc:654] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:21.439006 8612 tablet_bootstrap.cc:492] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: No bootstrap required, opened a new log
I20260929 15:45:21.439214 8612 ts_tablet_manager.cc:1397] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 15:45:21.440054 8612 raft_consensus.cc:348] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.440364 8612 raft_consensus.cc:374] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:21.440447 8612 raft_consensus.cc:729] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Initialized, Role: FOLLOWER
I20260929 15:45:21.440706 8612 consensus_queue.cc:260] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.442088 8612 ts_tablet_manager.cc:1428] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:45:21.444948 8613 tablet_bootstrap.cc:492] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: No bootstrap required, opened a new log
I20260929 15:45:21.445259 8613 ts_tablet_manager.cc:1397] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260929 15:45:21.445335 8614 tablet_bootstrap.cc:492] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: No bootstrap required, opened a new log
I20260929 15:45:21.445480 8614 ts_tablet_manager.cc:1397] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20260929 15:45:21.446362 8613 raft_consensus.cc:348] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.446357 8614 raft_consensus.cc:348] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.446631 8614 raft_consensus.cc:374] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:21.446723 8614 raft_consensus.cc:729] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3693251a3b4763bf9c941eb48454e2, State: Initialized, Role: FOLLOWER
I20260929 15:45:21.446842 8613 raft_consensus.cc:374] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:21.446995 8613 raft_consensus.cc:729] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Initialized, Role: FOLLOWER
I20260929 15:45:21.447018 8614 consensus_queue.cc:260] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.447674 8614 ts_tablet_manager.cc:1428] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:45:21.447422 8613 consensus_queue.cc:260] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.448609 8613 ts_tablet_manager.cc:1428] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:45:21.520649 8538 raft_consensus.cc:482] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:21.520977 8538 raft_consensus.cc:504] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.522888 8538 leader_election.cc:288] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501), b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941)
I20260929 15:45:21.541806 8537 raft_consensus.cc:482] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:21.542263 8537 raft_consensus.cc:504] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.542783 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.523693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54661 (local address 127.0.97.2:38941)
0929 15:45:21.523925 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.523931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.534906 (+ 10975us) server_negotiation.cc:407] Connection header received
0929 15:45:21.535181 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.535186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.535259 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.535322 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:21.536125 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.537316 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.539877 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.541175 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.541356 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.541493 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.542086 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.542489 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:45:21.542793 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.523129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38501 (local address 127.0.97.3:33479)
0929 15:45:21.534483 (+ 11354us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:21.534521 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:21.534743 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:21.535121 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:21.535125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:21.535152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:21.535447 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.535457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.537004 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.537009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.539637 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.539666 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.541802 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.541808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.542022 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.542065 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 15:45:21.542181 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:45:21.542372 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10859,"mutex_wait_us":46,"thread_start_us":11308,"threads_started":1}
I20260929 15:45:21.543118 8537 leader_election.cc:288] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:21.543711 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.523139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38941 (local address 127.0.97.3:54661)
0929 15:45:21.534782 (+ 11643us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:21.534801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:21.535013 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:21.535524 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:21.535528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:21.535552 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:21.535939 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.535952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.537508 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.537513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.539503 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.539536 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.541430 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.541437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.541624 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.541668 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 15:45:21.543089 (+ 1421us) client_negotiation.cc:240] Negotiation successful
0929 15:45:21.543272 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11513,"mutex_wait_us":79,"thread_start_us":10659,"threads_started":1}
I20260929 15:45:21.543715 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.523467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33479 (local address 127.0.97.1:38501)
0929 15:45:21.533903 (+ 10436us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.533911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.534679 (+ 768us) server_negotiation.cc:407] Connection header received
0929 15:45:21.534906 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.534919 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.534979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.535093 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:21.535622 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.536843 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.539930 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.541327 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.541480 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.541623 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.543141 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.543304 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10250,"thread_start_us":10073,"threads_started":1}
I20260929 15:45:21.544482 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "3d3693251a3b4763bf9c941eb48454e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13" is_pre_election: true
I20260929 15:45:21.544683 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2" is_pre_election: true
I20260929 15:45:21.544827 8317 raft_consensus.cc:2454] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3693251a3b4763bf9c941eb48454e2 in term 0.
I20260929 15:45:21.544929 8460 raft_consensus.cc:2454] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 0.
I20260929 15:45:21.545389 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "3d3693251a3b4763bf9c941eb48454e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" is_pre_election: true
I20260929 15:45:21.545317 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13" is_pre_election: true
I20260929 15:45:21.545639 8179 raft_consensus.cc:2454] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3693251a3b4763bf9c941eb48454e2 in term 0.
I20260929 15:45:21.545542 8388 leader_election.cc:302] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3693251a3b4763bf9c941eb48454e2, b134051cd2a74e9c9c314942d2b2db13; no voters:
I20260929 15:45:21.545657 8315 raft_consensus.cc:2454] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 0.
I20260929 15:45:21.546074 8538 raft_consensus.cc:2781] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:21.546099 8107 leader_election.cc:302] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, 3d3693251a3b4763bf9c941eb48454e2; no voters:
I20260929 15:45:21.546239 8538 raft_consensus.cc:482] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:21.546352 8538 raft_consensus.cc:3037] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.546401 8537 raft_consensus.cc:2781] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:21.546506 8537 raft_consensus.cc:482] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:21.546608 8537 raft_consensus.cc:3037] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.549731 8538 raft_consensus.cc:504] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.549731 8537 raft_consensus.cc:504] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.550616 8537 leader_election.cc:288] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 election: Requested vote from peers b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:21.551180 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2"
I20260929 15:45:21.551265 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13"
I20260929 15:45:21.551465 8315 raft_consensus.cc:3037] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.551506 8460 raft_consensus.cc:2379] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in current term 1: Already voted for candidate 3d3693251a3b4763bf9c941eb48454e2 in this term.
I20260929 15:45:21.552673 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "3d3693251a3b4763bf9c941eb48454e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13"
I20260929 15:45:21.553813 8538 leader_election.cc:288] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [CANDIDATE]: Term 1 election: Requested vote from peers 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501), b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941)
I20260929 15:45:21.554527 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed7c9a1d0e14cb79f5a024c748d4b11" candidate_uuid: "3d3693251a3b4763bf9c941eb48454e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd"
I20260929 15:45:21.554888 8179 raft_consensus.cc:2379] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d3693251a3b4763bf9c941eb48454e2 in current term 1: Already voted for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in this term.
I20260929 15:45:21.555508 8386 leader_election.cc:302] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d3693251a3b4763bf9c941eb48454e2; no voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, b134051cd2a74e9c9c314942d2b2db13
I20260929 15:45:21.555737 8315 raft_consensus.cc:2454] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 1.
I20260929 15:45:21.556102 8538 raft_consensus.cc:2726] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 15:45:21.556416 8108 leader_election.cc:302] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, b134051cd2a74e9c9c314942d2b2db13; no voters: 3d3693251a3b4763bf9c941eb48454e2
I20260929 15:45:21.556854 8537 raft_consensus.cc:2781] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:21.557654 8537 raft_consensus.cc:686] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Running, Role: LEADER
I20260929 15:45:21.558035 8537 consensus_queue.cc:237] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.561677 8019 catalog_manager.cc:5495] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 15:45:21.597472 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.591197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.0.97.1:38501)
0929 15:45:21.591428 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.591434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.591454 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:45:21.591629 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.591635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.591766 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.591853 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:21.592293 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.593538 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.595456 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.595902 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.596053 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.596583 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.596730 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.596838 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.597019 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.597120 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:45:21.600919 8460 raft_consensus.cc:1261] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Refusing update from remote peer 1d23bbf1b5774e52ab4ad0b33b0541bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.600934 8315 raft_consensus.cc:1261] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Refusing update from remote peer 1d23bbf1b5774e52ab4ad0b33b0541bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.601689 8621 consensus_queue.cc:1035] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.602018 8537 consensus_queue.cc:1035] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.621137 8019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:21.621439 8019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:21.624969 8019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Sending DeleteTablet for 3 replicas of tablet 4ed7c9a1d0e14cb79f5a024c748d4b11
I20260929 15:45:21.625926 8156 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54336
I20260929 15:45:21.625962 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50940
I20260929 15:45:21.626240 8598 tablet_replica.cc:330] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: stopping tablet replica
I20260929 15:45:21.626315 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260929 15:45:21.626437 8598 raft_consensus.cc:2229] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:21.626664 8597 tablet_replica.cc:330] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: stopping tablet replica
I20260929 15:45:21.626881 8597 raft_consensus.cc:2229] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:21.627125 8597 raft_consensus.cc:2258] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:21.627105 8598 raft_consensus.cc:2258] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:21.627411 8596 tablet_replica.cc:330] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: stopping tablet replica
I20260929 15:45:21.627616 8596 raft_consensus.cc:2229] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:21.627861 8596 raft_consensus.cc:2258] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:21.628517 8598 ts_tablet_manager.cc:1905] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:21.629102 8596 ts_tablet_manager.cc:1905] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:21.630205 8019 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:21.631206 8019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:21.635648 8598 ts_tablet_manager.cc:1918] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:21.635670 8596 ts_tablet_manager.cc:1918] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:21.635839 8596 log.cc:1196] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/4ed7c9a1d0e14cb79f5a024c748d4b11
I20260929 15:45:21.635895 8598 log.cc:1196] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/4ed7c9a1d0e14cb79f5a024c748d4b11
I20260929 15:45:21.636336 8598 ts_tablet_manager.cc:1939] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting consensus metadata
I20260929 15:45:21.636613 8596 ts_tablet_manager.cc:1939] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P 3d3693251a3b4763bf9c941eb48454e2: Deleting consensus metadata
I20260929 15:45:21.637660 8003 catalog_manager.cc:4841] TS 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501): tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 (table test-table [id=19864f9c7c0140669a66f6b65ad2b960]) successfully deleted
I20260929 15:45:21.637811 8004 catalog_manager.cc:4841] TS 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333): tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 (table test-table [id=19864f9c7c0140669a66f6b65ad2b960]) successfully deleted
I20260929 15:45:21.638842 8597 ts_tablet_manager.cc:1905] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:21.640890 8436 tablet_service.cc:1467] Processing CreateTablet for tablet c5d12abc19dc4322b7bcbbff471d0f86 (DEFAULT_TABLE table=test-table [id=630c20abca57456fb94234a48f092386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:21.641389 8436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d12abc19dc4322b7bcbbff471d0f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:21.641345 8295 tablet_service.cc:1467] Processing CreateTablet for tablet c5d12abc19dc4322b7bcbbff471d0f86 (DEFAULT_TABLE table=test-table [id=630c20abca57456fb94234a48f092386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:21.641744 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d12abc19dc4322b7bcbbff471d0f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:21.642163 8156 tablet_service.cc:1467] Processing CreateTablet for tablet c5d12abc19dc4322b7bcbbff471d0f86 (DEFAULT_TABLE table=test-table [id=630c20abca57456fb94234a48f092386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:21.642601 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d12abc19dc4322b7bcbbff471d0f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:21.648837 8613 tablet_bootstrap.cc:492] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Bootstrap starting.
I20260929 15:45:21.651029 8614 tablet_bootstrap.cc:492] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Bootstrap starting.
I20260929 15:45:21.651077 8613 tablet_bootstrap.cc:654] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:21.653191 8614 tablet_bootstrap.cc:654] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:21.653764 8597 ts_tablet_manager.cc:1918] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:21.653935 8597 log.cc:1196] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/4ed7c9a1d0e14cb79f5a024c748d4b11
I20260929 15:45:21.654430 8597 ts_tablet_manager.cc:1939] T 4ed7c9a1d0e14cb79f5a024c748d4b11 P b134051cd2a74e9c9c314942d2b2db13: Deleting consensus metadata
I20260929 15:45:21.655875 8005 catalog_manager.cc:4841] TS b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941): tablet 4ed7c9a1d0e14cb79f5a024c748d4b11 (table test-table [id=19864f9c7c0140669a66f6b65ad2b960]) successfully deleted
I20260929 15:45:21.657486 8612 tablet_bootstrap.cc:492] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Bootstrap starting.
I20260929 15:45:21.659360 8612 tablet_bootstrap.cc:654] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:21.661580 8613 tablet_bootstrap.cc:492] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: No bootstrap required, opened a new log
I20260929 15:45:21.661875 8613 ts_tablet_manager.cc:1397] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260929 15:45:21.663378 8613 raft_consensus.cc:348] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.663805 8613 raft_consensus.cc:374] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:21.663931 8613 raft_consensus.cc:729] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b134051cd2a74e9c9c314942d2b2db13, State: Initialized, Role: FOLLOWER
I20260929 15:45:21.664281 8613 consensus_queue.cc:260] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.666483 8613 ts_tablet_manager.cc:1428] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 15:45:21.669358 8612 tablet_bootstrap.cc:492] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: No bootstrap required, opened a new log
I20260929 15:45:21.669565 8612 ts_tablet_manager.cc:1397] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 15:45:21.670428 8612 raft_consensus.cc:348] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.670694 8612 raft_consensus.cc:374] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:21.670763 8612 raft_consensus.cc:729] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Initialized, Role: FOLLOWER
I20260929 15:45:21.671315 8612 consensus_queue.cc:260] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.672776 8612 ts_tablet_manager.cc:1428] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 15:45:21.674928 8614 tablet_bootstrap.cc:492] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: No bootstrap required, opened a new log
I20260929 15:45:21.675734 8614 ts_tablet_manager.cc:1397] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.009s
I20260929 15:45:21.676822 8614 raft_consensus.cc:348] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.677114 8614 raft_consensus.cc:374] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:21.677214 8614 raft_consensus.cc:729] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3693251a3b4763bf9c941eb48454e2, State: Initialized, Role: FOLLOWER
I20260929 15:45:21.677466 8614 consensus_queue.cc:260] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.678355 8614 ts_tablet_manager.cc:1428] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 15:45:21.833765 8621 raft_consensus.cc:482] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:21.834151 8621 raft_consensus.cc:504] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.835240 8621 leader_election.cc:288] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:21.835968 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d12abc19dc4322b7bcbbff471d0f86" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13" is_pre_election: true
I20260929 15:45:21.836027 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d12abc19dc4322b7bcbbff471d0f86" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2" is_pre_election: true
I20260929 15:45:21.836362 8315 raft_consensus.cc:2454] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 0.
I20260929 15:45:21.836433 8460 raft_consensus.cc:2454] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 0.
I20260929 15:45:21.837011 8107 leader_election.cc:302] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, 3d3693251a3b4763bf9c941eb48454e2, b134051cd2a74e9c9c314942d2b2db13; no voters:
I20260929 15:45:21.837416 8621 raft_consensus.cc:2781] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:21.837545 8621 raft_consensus.cc:482] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:21.837693 8621 raft_consensus.cc:3037] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.840076 8621 raft_consensus.cc:504] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.840921 8621 leader_election.cc:288] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 election: Requested vote from peers b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941), 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333)
I20260929 15:45:21.841585 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d12abc19dc4322b7bcbbff471d0f86" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3693251a3b4763bf9c941eb48454e2"
I20260929 15:45:21.841643 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d12abc19dc4322b7bcbbff471d0f86" candidate_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b134051cd2a74e9c9c314942d2b2db13"
I20260929 15:45:21.841948 8460 raft_consensus.cc:3037] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.842002 8315 raft_consensus.cc:3037] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:21.844524 8460 raft_consensus.cc:2454] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 1.
I20260929 15:45:21.845188 8107 leader_election.cc:302] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d23bbf1b5774e52ab4ad0b33b0541bd, 3d3693251a3b4763bf9c941eb48454e2; no voters:
I20260929 15:45:21.845527 8315 raft_consensus.cc:2454] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d23bbf1b5774e52ab4ad0b33b0541bd in term 1.
I20260929 15:45:21.845769 8621 raft_consensus.cc:2781] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:21.846063 8621 raft_consensus.cc:686] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 LEADER]: Becoming Leader. State: Replica: 1d23bbf1b5774e52ab4ad0b33b0541bd, State: Running, Role: LEADER
I20260929 15:45:21.846441 8621 consensus_queue.cc:237] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 15:45:21.851110 8018 catalog_manager.cc:5495] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d23bbf1b5774e52ab4ad0b33b0541bd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 15:45:21.901032 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.891787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59841 (local address 127.0.97.2:38941)
0929 15:45:21.892184 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.892191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.893181 (+ 990us) server_negotiation.cc:407] Connection header received
0929 15:45:21.893600 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.893605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.893712 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.893824 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:21.894718 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.895637 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.897954 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.898817 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.898944 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.899783 (+ 839us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.900088 (+ 305us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.900296 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.900532 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.900640 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 15:45:21.901216 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.891688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38941 (local address 127.0.97.1:59841)
0929 15:45:21.893003 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:21.893033 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:21.893322 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:21.894055 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:21.894063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:21.894086 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:45:21.894471 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.894484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.895868 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.895876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.897733 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.897754 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.899066 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.899075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.899290 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.899402 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.900305 (+ 903us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.900344 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:45:21.900495 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 15:45:21.900732 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":714,"threads_started":1}
I20260929 15:45:21.912876 8460 raft_consensus.cc:1261] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Refusing update from remote peer 1d23bbf1b5774e52ab4ad0b33b0541bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.913028 8315 raft_consensus.cc:1261] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Refusing update from remote peer 1d23bbf1b5774e52ab4ad0b33b0541bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:21.913755 8537 consensus_queue.cc:1035] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3693251a3b4763bf9c941eb48454e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.914147 8621 consensus_queue.cc:1035] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b134051cd2a74e9c9c314942d2b2db13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:21.954787 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.946694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38501 (local address 127.0.97.2:52709)
0929 15:45:21.947488 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:21.947507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:21.947726 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:21.948287 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:21.948292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:21.948307 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:45:21.948665 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.948677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.950176 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.950184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.952116 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:21.952135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.953196 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.953203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:21.953408 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.953464 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.953949 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.953963 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:45:21.954055 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 15:45:21.954232 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":459,"threads_started":1}
I20260929 15:45:21.954790 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:21.946838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52709 (local address 127.0.97.1:38501)
0929 15:45:21.947028 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:21.947035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:21.947606 (+ 571us) server_negotiation.cc:407] Connection header received
0929 15:45:21.947893 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:21.947899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:21.947987 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:21.948052 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:21.948854 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.949992 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.952332 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:21.952964 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:21.953131 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:21.953661 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:21.953827 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:21.953943 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:21.954268 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 15:45:21.954413 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:45:22.001880 8018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:22.002166 8018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:22.005913 8018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7674b1f0d843449f8e592dc3d8f53a06: Sending DeleteTablet for 3 replicas of tablet c5d12abc19dc4322b7bcbbff471d0f86
I20260929 15:45:22.006910 8156 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d12abc19dc4322b7bcbbff471d0f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54336
I20260929 15:45:22.007293 8436 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d12abc19dc4322b7bcbbff471d0f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260929 15:45:22.007627 8598 tablet_replica.cc:330] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: stopping tablet replica
I20260929 15:45:22.007825 8598 raft_consensus.cc:2229] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:22.008328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 8090
I20260929 15:45:22.008350 8596 tablet_replica.cc:330] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: stopping tablet replica
I20260929 15:45:22.008543 8596 raft_consensus.cc:2229] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:22.008769 8596 raft_consensus.cc:2258] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:22.010063 8598 raft_consensus.cc:2258] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:22.010483 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d12abc19dc4322b7bcbbff471d0f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50940
I20260929 15:45:22.010673 8596 ts_tablet_manager.cc:1905] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:22.010807 8597 tablet_replica.cc:330] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: stopping tablet replica
I20260929 15:45:22.010993 8597 raft_consensus.cc:2229] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:22.011227 8597 raft_consensus.cc:2258] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:22.017143 8598 ts_tablet_manager.cc:1905] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:22.017174 8596 ts_tablet_manager.cc:1918] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:45:22.017378 8596 log.cc:1196] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/c5d12abc19dc4322b7bcbbff471d0f86
I20260929 15:45:22.018069 8596 ts_tablet_manager.cc:1939] T c5d12abc19dc4322b7bcbbff471d0f86 P 3d3693251a3b4763bf9c941eb48454e2: Deleting consensus metadata
I20260929 15:45:22.019857 8004 catalog_manager.cc:4841] TS 3d3693251a3b4763bf9c941eb48454e2 (127.0.97.3:36333): tablet c5d12abc19dc4322b7bcbbff471d0f86 (table test-table [id=630c20abca57456fb94234a48f092386]) successfully deleted
I20260929 15:45:22.023321 8597 ts_tablet_manager.cc:1905] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:22.026881 8598 ts_tablet_manager.cc:1918] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:45:22.027050 8598 log.cc:1196] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/c5d12abc19dc4322b7bcbbff471d0f86
I20260929 15:45:22.027495 8598 ts_tablet_manager.cc:1939] T c5d12abc19dc4322b7bcbbff471d0f86 P 1d23bbf1b5774e52ab4ad0b33b0541bd: Deleting consensus metadata
Sep 29 15:45:22 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 15:45:22.028741 8003 catalog_manager.cc:4841] TS 1d23bbf1b5774e52ab4ad0b33b0541bd (127.0.97.1:38501): tablet c5d12abc19dc4322b7bcbbff471d0f86 (table test-table [id=630c20abca57456fb94234a48f092386]) successfully deleted
I20260929 15:45:22.032267 8597 ts_tablet_manager.cc:1918] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:45:22.032541 8597 log.cc:1196] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/c5d12abc19dc4322b7bcbbff471d0f86
I20260929 15:45:22.033066 8597 ts_tablet_manager.cc:1939] T c5d12abc19dc4322b7bcbbff471d0f86 P b134051cd2a74e9c9c314942d2b2db13: Deleting consensus metadata
I20260929 15:45:22.034420 8005 catalog_manager.cc:4841] TS b134051cd2a74e9c9c314942d2b2db13 (127.0.97.2:38941): tablet c5d12abc19dc4322b7bcbbff471d0f86 (table test-table [id=630c20abca57456fb94234a48f092386]) successfully deleted
I20260929 15:45:22.037885 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:22.009864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.0.97.1:38501)
0929 15:45:22.016243 (+ 6379us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:22.016252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:22.016284 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:45:22.016364 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:22.016372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:22.016533 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:22.016682 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:22.018770 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:22.019955 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:22.022182 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:22.022971 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:22.023146 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:22.028983 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:22.029004 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:22.029027 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:22.029102 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:22.033636 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:22.034836 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:22.034847 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:22.034850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:22.035050 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:22.035658 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:22.035662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:22.035665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:22.036388 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:22.036776 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:22.037176 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 15:45:22.037436 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6156}
I20260929 15:45:22.038932 8221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:22.200064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 8090
I20260929 15:45:22.228394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 8230
Sep 29 15:45:22 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 15:45:22.247274 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:22.229348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.97.2:38941)
0929 15:45:22.229539 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:22.229544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:22.229582 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:45:22.229642 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:22.229646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:22.229731 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:22.229826 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:22.231181 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:22.231939 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:22.233607 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:22.234176 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:22.234390 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:22.239830 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:22.239853 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:22.239876 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:22.239941 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:22.243558 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:22.244588 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:22.244599 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:22.244603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:22.244820 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:22.245242 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:22.245246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:22.245248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:22.245915 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:22.246267 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:22.246688 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 15:45:22.246975 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:45:22.248322 8361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:22.410382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 8230
I20260929 15:45:22.439419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 8370
Sep 29 15:45:22 dist-test-slave-7491 krb5kdc[7974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 15:45:22.460080 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:22.440316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.0.97.3:36333)
0929 15:45:22.440793 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:22.440798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:22.440815 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:45:22.440870 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:22.440873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:22.440944 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:22.441056 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:22.442527 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:22.443263 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:22.445219 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:22.445697 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:22.445843 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:22.451383 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:22.451401 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:22.451420 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:22.451467 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:22.455392 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:22.456818 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:22.456833 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:22.456840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:22.457078 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:22.457874 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:22.457881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:22.457884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:22.458447 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:22.458788 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:22.459299 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 15:45:22.459692 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":284,"threads_started":1}
I20260929 15:45:22.461186 8501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:22.576467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 8370
I20260929 15:45:22.595656 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 7987
I20260929 15:45:22.597049 8059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:22.722354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 7987
2026-09-29T15:45:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6743 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:45:22 dist-test-slave-7491 krb5kdc[8650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:45:22 dist-test-slave-7491 krb5kdc[8650](info): set up 2 sockets
Sep 29 15:45:22 dist-test-slave-7491 krb5kdc[8650](info): commencing operation
krb5kdc: starting...
W20260929 15:45:23.897758 388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:45:24 dist-test-slave-7491 krb5kdc[8650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:45:24.166414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43651
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:24.571877 8663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:24.572343 8663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:24.572512 8663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:24.581177 8663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:24.581372 8663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:24.581488 8663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:24.581601 8663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:24.581686 8663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:24.592563 8663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:24.594095 8663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:24.595606 8663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:24.611300 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:24.611281 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:24.611203 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:24.613708 8663 server_base.cc:1034] running on GCE node
I20260929 15:45:24.615080 8663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:24.617604 8663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:24.618981 8663 hybrid_clock.cc:648] HybridClock initialized: now 1790696724618905 us; error 99 us; skew 500 ppm
I20260929 15:45:24.619585 8663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T15:45:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1965 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:45:24 dist-test-slave-7491 krb5kdc[8677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:45:24 dist-test-slave-7491 krb5kdc[8677](info): set up 2 sockets
Sep 29 15:45:24 dist-test-slave-7491 krb5kdc[8677](info): commencing operation
krb5kdc: starting...
W20260929 15:45:25.857316 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:45:26 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:45:26.132946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46543
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:26.552331 8690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:26.552762 8690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:26.552923 8690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:26.562610 8690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:26.562835 8690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:26.562954 8690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:26.563062 8690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:26.563146 8690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:26.574945 8690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:26.577169 8690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:26.579062 8690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:26.594831 8696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:26.594925 8698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:26.594926 8695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:26.595638 8690 server_base.cc:1034] running on GCE node
I20260929 15:45:26.596829 8690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:26.599018 8690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:26.600436 8690 hybrid_clock.cc:648] HybridClock initialized: now 1790696726600327 us; error 72 us; skew 500 ppm
I20260929 15:45:26.601090 8690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:26 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:26.611028 8690 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:45:26.613130 8690 webserver.cc:460] Webserver started at http://127.0.97.62:34435/ using document root <none> and password file <none>
I20260929 15:45:26.613986 8690 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:26.614115 8690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:26.614871 8690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:26.619112 8690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "5ff86922ce154780aa16ae20ab1d1cd3"
format_stamp: "Formatted at 2026-09-29 15:45:26 on dist-test-slave-7491"
server_key: "816bd726d5f1fed1e8f9778db68ffc19"
server_key_iv: "b1b3c0f18af83a75047fc362f936d313"
server_key_version: "encryptionkey@0"
I20260929 15:45:26.620139 8690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ff86922ce154780aa16ae20ab1d1cd3"
format_stamp: "Formatted at 2026-09-29 15:45:26 on dist-test-slave-7491"
server_key: "816bd726d5f1fed1e8f9778db68ffc19"
server_key_iv: "b1b3c0f18af83a75047fc362f936d313"
server_key_version: "encryptionkey@0"
I20260929 15:45:26.627502 8690 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 15:45:26.632261 8705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:26.634402 8690 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:26.634691 8690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "5ff86922ce154780aa16ae20ab1d1cd3"
format_stamp: "Formatted at 2026-09-29 15:45:26 on dist-test-slave-7491"
server_key: "816bd726d5f1fed1e8f9778db68ffc19"
server_key_iv: "b1b3c0f18af83a75047fc362f936d313"
server_key_version: "encryptionkey@0"
I20260929 15:45:26.634959 8690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:26.661412 8690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:26.666874 8690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:26.667237 8690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:26.690559 8690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46543
I20260929 15:45:26.690611 8756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46543 every 8 connection(s)
I20260929 15:45:26.692212 8690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:26.695312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 8690
I20260929 15:45:26.695582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:26.696483 388 external_mini_cluster.cc:1428] Setting key ab41fd0cffdbd4fbc2d35da79ca5d633
I20260929 15:45:26.697254 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:26.714448 8757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Bootstrap starting.
Sep 29 15:45:26 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:26.719197 8757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:26.720839 8757 log.cc:824] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:26.725045 8757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: No bootstrap required, opened a new log
I20260929 15:45:26.726203 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:26.699249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.0.97.62:46543)
0929 15:45:26.700349 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:26.700364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:26.700426 (+ 62us) server_negotiation.cc:407] Connection header received
0929 15:45:26.702990 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:26.703027 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:26.703608 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:26.704051 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:26.706312 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:26.707712 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:26.709734 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:26.710839 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:26.711037 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:26.716838 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:26.716852 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:26.716876 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:26.716955 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:26.720873 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:26.721848 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:26.721855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:26.721864 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:26.722121 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:26.722558 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:26.722568 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:26.722570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:26.723258 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:26.723607 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:26.724121 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 15:45:26.724766 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":508,"threads_started":1}
I20260929 15:45:26.734743 8757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:26.735451 8757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:26.735654 8757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff86922ce154780aa16ae20ab1d1cd3, State: Initialized, Role: FOLLOWER
I20260929 15:45:26.736477 8757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:26.736760 8757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:26.736876 8757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:26.737017 8757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:26.739602 8757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:26.740312 8757 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff86922ce154780aa16ae20ab1d1cd3; no voters:
I20260929 15:45:26.741101 8757 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:26.741472 8762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:26.742404 8762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff86922ce154780aa16ae20ab1d1cd3, State: Running, Role: LEADER
I20260929 15:45:26.743149 8762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:26.743672 8757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:26.747152 8763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } } }
I20260929 15:45:26.747684 8763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:26.748122 8764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff86922ce154780aa16ae20ab1d1cd3. Latest consensus state: current_term: 1 leader_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } } }
I20260929 15:45:26.748997 8764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:26.749873 8769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:26.755879 8769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:26.766701 8769 catalog_manager.cc:1347] Generated new cluster ID: 8c16549676284bf681b049f24e9284b7
I20260929 15:45:26.766949 8769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:26.811643 8769 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:26.813395 8769 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:26.823431 8769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Generated new TSK 0
I20260929 15:45:26.824958 8769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:45:26.898490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46543
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:27.306425 8783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:27.306768 8783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:27.306927 8783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:27.314723 8783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:27.314868 8783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:27.315062 8783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:27.325629 8783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:27.327195 8783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:27.328871 8783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:27.344801 8788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:27.344758 8791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:27.344713 8789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:27.346087 8783 server_base.cc:1034] running on GCE node
I20260929 15:45:27.346843 8783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:27.347961 8783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:27.349256 8783 hybrid_clock.cc:648] HybridClock initialized: now 1790696727349234 us; error 71 us; skew 500 ppm
I20260929 15:45:27.349665 8783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:27 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:27.354362 8783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:45:27.356266 8783 webserver.cc:460] Webserver started at http://127.0.97.1:33577/ using document root <none> and password file <none>
I20260929 15:45:27.357002 8783 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:27.357111 8783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:27.357383 8783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:27.360695 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "5140adfa69b84fa5bcfc8691674967b4"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "94c3e97a4535d376c28b3762c6ed37db"
server_key_iv: "bb16de39feb629c38cc60e073c1c50b3"
server_key_version: "encryptionkey@0"
I20260929 15:45:27.361532 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5140adfa69b84fa5bcfc8691674967b4"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "94c3e97a4535d376c28b3762c6ed37db"
server_key_iv: "bb16de39feb629c38cc60e073c1c50b3"
server_key_version: "encryptionkey@0"
I20260929 15:45:27.367686 8783 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:45:27.372865 8798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:27.375068 8783 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:45:27.375329 8783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5140adfa69b84fa5bcfc8691674967b4"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "94c3e97a4535d376c28b3762c6ed37db"
server_key_iv: "bb16de39feb629c38cc60e073c1c50b3"
server_key_version: "encryptionkey@0"
I20260929 15:45:27.375582 8783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:27.400724 8783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:27.405720 8783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:27.406008 8783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:27.407043 8783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:27.408571 8783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:27.408824 8783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:27.409006 8783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:27.409053 8783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:27.448146 8783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37183
I20260929 15:45:27.448329 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37183 every 8 connection(s)
I20260929 15:45:27.449613 8783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:27.451328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 8783
I20260929 15:45:27.451584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:27.452397 388 external_mini_cluster.cc:1428] Setting key bee9c3506f1ff95ce8a11d48ecc71df1
Sep 29 15:45:27 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:27.477941 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:27.453119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47007 (local address 127.0.97.62:46543)
0929 15:45:27.453610 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:27.453617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:27.454985 (+ 1368us) server_negotiation.cc:407] Connection header received
0929 15:45:27.458131 (+ 3146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:27.458135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:27.458212 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:27.458336 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:27.460940 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:27.461884 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:27.464410 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:27.464997 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:27.465136 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:27.470134 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:27.470144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:27.470152 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:27.470207 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:27.474243 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:27.475291 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:27.475304 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:27.475307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:27.475459 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:27.475962 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:27.475974 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:27.475978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:27.476352 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:27.476518 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:27.477294 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 15:45:27.477553 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260929 15:45:27.479044 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:27.453806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:47007)
0929 15:45:27.454784 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:27.454851 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:27.457733 (+ 2882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:27.458689 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:27.458704 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:27.459605 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:27.460566 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:27.460606 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:27.462168 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:27.462176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:27.464092 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:27.464109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:27.465295 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:27.465306 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:27.465666 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:27.466578 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:27.466616 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:27.469743 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:27.474514 (+ 4771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:27.474523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:27.474549 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:27.475093 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:27.475612 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:27.475617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:27.475619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:27.475825 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:27.476494 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:27.476503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:27.476907 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 15:45:27.477199 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 15:45:27.477815 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":372,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:45:27.481665 8911 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46543
I20260929 15:45:27.482074 8911 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:45:27.483000 8911 heartbeater.cc:511] Master 127.0.97.62:46543 requested a full tablet report, sending...
I20260929 15:45:27.486208 8722 ts_manager.cc:194] Registered new tserver with Master: 5140adfa69b84fa5bcfc8691674967b4 (127.0.97.1:37183)
I20260929 15:45:27.489051 8722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:45:27.516552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46543
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:27.879519 8917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:27.879912 8917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:27.880069 8917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:27.888072 8917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:27.888309 8917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:27.888441 8917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:27.898897 8917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:27.900348 8917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:27.901829 8917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:27.917418 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:27.917276 8923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:27.917582 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:27.918406 8917 server_base.cc:1034] running on GCE node
I20260929 15:45:27.919222 8917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:27.920461 8917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:27.921767 8917 hybrid_clock.cc:648] HybridClock initialized: now 1790696727921709 us; error 76 us; skew 500 ppm
I20260929 15:45:27.922156 8917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:27 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:27.926363 8917 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:45:27.928813 8917 webserver.cc:460] Webserver started at http://127.0.97.2:44715/ using document root <none> and password file <none>
I20260929 15:45:27.929912 8917 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:27.930102 8917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:27.930565 8917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:27.934810 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "9860f269b34d4699a0df356ef3d75a39"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "f9dbbcd88b92edb5274055869c4140d4"
server_key_iv: "03230255a5192266259f8544ba876fe8"
server_key_version: "encryptionkey@0"
I20260929 15:45:27.936389 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9860f269b34d4699a0df356ef3d75a39"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "f9dbbcd88b92edb5274055869c4140d4"
server_key_iv: "03230255a5192266259f8544ba876fe8"
server_key_version: "encryptionkey@0"
I20260929 15:45:27.943593 8917 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 15:45:27.949198 8932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:27.951789 8917 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:45:27.952045 8917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "9860f269b34d4699a0df356ef3d75a39"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "f9dbbcd88b92edb5274055869c4140d4"
server_key_iv: "03230255a5192266259f8544ba876fe8"
server_key_version: "encryptionkey@0"
I20260929 15:45:27.952467 8917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:27.985183 8917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:27.992875 8917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:27.993229 8917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:27.994735 8917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:27.997470 8917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:27.997704 8917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:45:27.997869 8917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:27.997956 8917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:28.040901 8917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45107
I20260929 15:45:28.040951 9044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45107 every 8 connection(s)
I20260929 15:45:28.042871 8917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:28.044922 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 8917
I20260929 15:45:28.045153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:28.045662 388 external_mini_cluster.cc:1428] Setting key d3f196f2a1b8c79f0d6a7facb66b6afe
Sep 29 15:45:28 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:28.069756 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:28.047156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53035 (local address 127.0.97.62:46543)
0929 15:45:28.047721 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:28.047728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:28.048842 (+ 1114us) server_negotiation.cc:407] Connection header received
0929 15:45:28.050936 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:28.050941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:28.051038 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:28.051181 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:28.053095 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.053851 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.055749 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.056278 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.056424 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:28.061509 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:28.061515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:28.061519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:28.061564 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:28.065088 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:28.066544 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:28.066549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:28.066552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:28.066752 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:28.067483 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:28.067490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:28.067492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:28.067864 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:28.068008 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:28.069256 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0929 15:45:28.069472 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260929 15:45:28.072229 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:28.047681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:53035)
0929 15:45:28.048664 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:28.048719 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:28.050561 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:28.051463 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:28.051473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:28.052153 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:28.052879 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:28.052903 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.054036 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.054040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:28.055551 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:28.055566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.056550 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.056558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:28.056813 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:28.057651 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:28.057677 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:28.061062 (+ 3385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:28.065379 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:28.065395 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:28.065422 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:28.066286 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:28.066909 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:28.066916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:28.066923 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:28.067234 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:28.068036 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:28.068048 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:28.068659 (+ 611us) client_negotiation.cc:769] Sending connection context
0929 15:45:28.069020 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 15:45:28.069717 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":441,"threads_started":1}
I20260929 15:45:28.074386 9045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46543
I20260929 15:45:28.074780 9045 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:28.075641 9045 heartbeater.cc:511] Master 127.0.97.62:46543 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:45:28.077713 8722 ts_manager.cc:194] Registered new tserver with Master: 9860f269b34d4699a0df356ef3d75a39 (127.0.97.2:45107)
I20260929 15:45:28.078691 8722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:45:28.112636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46543
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:28.493453 8911 heartbeater.cc:503] Master 127.0.97.62:46543 was elected leader, sending a full tablet report...
W20260929 15:45:28.522573 9051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:28.523015 9051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:28.523222 9051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:28.534801 9051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:28.534957 9051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:28.535173 9051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:28.548609 9051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:28.550145 9051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:28.551664 9051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:28.569958 9059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:28.570091 9057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:28.570060 9056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:28.571165 9051 server_base.cc:1034] running on GCE node
I20260929 15:45:28.571949 9051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:28.573230 9051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:28.574641 9051 hybrid_clock.cc:648] HybridClock initialized: now 1790696728574551 us; error 50 us; skew 500 ppm
I20260929 15:45:28.575222 9051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:28 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:28.579584 9051 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:45:28.581889 9051 webserver.cc:460] Webserver started at http://127.0.97.3:34187/ using document root <none> and password file <none>
I20260929 15:45:28.582762 9051 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:28.582959 9051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:28.583302 9051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:28.586700 9051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "0c786009d113499bacbb17965098e3dd"
format_stamp: "Formatted at 2026-09-29 15:45:28 on dist-test-slave-7491"
server_key: "8f5b5217780cf83a1d3c766bd53da580"
server_key_iv: "e4967594c25ae2b0c34b61216a9df7a1"
server_key_version: "encryptionkey@0"
I20260929 15:45:28.587654 9051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0c786009d113499bacbb17965098e3dd"
format_stamp: "Formatted at 2026-09-29 15:45:28 on dist-test-slave-7491"
server_key: "8f5b5217780cf83a1d3c766bd53da580"
server_key_iv: "e4967594c25ae2b0c34b61216a9df7a1"
server_key_version: "encryptionkey@0"
I20260929 15:45:28.594105 9051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 15:45:28.599527 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:28.601976 9051 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 15:45:28.602249 9051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "0c786009d113499bacbb17965098e3dd"
format_stamp: "Formatted at 2026-09-29 15:45:28 on dist-test-slave-7491"
server_key: "8f5b5217780cf83a1d3c766bd53da580"
server_key_iv: "e4967594c25ae2b0c34b61216a9df7a1"
server_key_version: "encryptionkey@0"
I20260929 15:45:28.602519 9051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:28.624660 9051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:28.631261 9051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:28.631632 9051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:28.632921 9051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:28.634496 9051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:28.634609 9051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:28.634766 9051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:28.634840 9051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:28.677211 9051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36207
I20260929 15:45:28.677372 9178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36207 every 8 connection(s)
I20260929 15:45:28.679224 9051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:28.682818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9051
I20260929 15:45:28.683063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:28.683594 388 external_mini_cluster.cc:1428] Setting key a571783d5226d21037165c41ff178faa
Sep 29 15:45:28 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:28.708216 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:28.683080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41291 (local address 127.0.97.62:46543)
0929 15:45:28.683778 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:28.683787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:28.685669 (+ 1882us) server_negotiation.cc:407] Connection header received
0929 15:45:28.687947 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:28.687952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:28.688028 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:28.688128 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:28.690631 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.692102 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.694090 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.694758 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.694888 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:28.700393 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:28.700401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:28.700404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:28.700449 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:28.704071 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:28.705146 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:28.705151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:28.705155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:28.705297 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:28.705878 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:28.705882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:28.705884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:28.706292 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:28.706458 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:28.707404 (+ 946us) server_negotiation.cc:299] Negotiation successful
0929 15:45:28.707714 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":421,"threads_started":1}
I20260929 15:45:28.709676 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:28.683962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:41291)
0929 15:45:28.685273 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:28.685330 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:28.687540 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:28.688685 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:28.688698 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:28.689568 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:28.690359 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:28.690386 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.692348 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.692353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:28.693904 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:28.693920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.695058 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.695064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:28.695286 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:28.696398 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:28.696443 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:28.700006 (+ 3563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:28.704390 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:28.704401 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:28.704423 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:28.704987 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:28.705441 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:28.705447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:28.705451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:28.705727 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:28.706427 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:28.706438 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:28.706958 (+ 520us) client_negotiation.cc:769] Sending connection context
0929 15:45:28.707342 (+ 384us) client_negotiation.cc:240] Negotiation successful
0929 15:45:28.708037 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":630,"threads_started":1}
I20260929 15:45:28.711802 9179 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46543
I20260929 15:45:28.712352 9179 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:28.713543 9179 heartbeater.cc:511] Master 127.0.97.62:46543 requested a full tablet report, sending...
I20260929 15:45:28.715610 8722 ts_manager.cc:194] Registered new tserver with Master: 0c786009d113499bacbb17965098e3dd (127.0.97.3:36207)
I20260929 15:45:28.716547 8722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41291
I20260929 15:45:28.727787 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 15:45:28.892895 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 8783
Sep 29 15:45:28 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:45:28.915371 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:28.894375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.97.1:37183)
0929 15:45:28.895139 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:28.895147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:28.895182 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:45:28.895255 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:28.895263 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:28.895493 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:28.895690 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:28.897597 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.898704 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.900455 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:28.901219 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:28.901371 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:28.906713 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:28.906732 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:28.906751 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:28.906804 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:28.911199 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:28.912617 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:28.912629 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:28.912633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:28.912848 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:28.913375 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:28.913379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:28.913382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:28.913805 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:28.914220 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:28.914663 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 15:45:28.914961 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":412,"threads_started":1}
I20260929 15:45:28.917215 8906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:29.007153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 8783
I20260929 15:45:29.027132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 8917
Sep 29 15:45:29 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:45:29.047741 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:29.028224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.0.97.2:45107)
0929 15:45:29.028875 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:29.028884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:29.028911 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:45:29.028981 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:29.028988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:29.029226 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:29.029362 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:29.030687 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:29.031852 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:29.033434 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:29.034041 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:29.034153 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:29.040352 (+ 6199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:29.040374 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:29.040395 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:29.040450 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:29.044431 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:29.045318 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:29.045327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:29.045330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:29.045516 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:29.045971 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:29.045977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:29.045981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:29.046352 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:29.046670 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:29.047058 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 15:45:29.047342 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":343,"threads_started":1}
I20260929 15:45:29.049177 9040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:29.144454 9045 heartbeater.cc:503] Master 127.0.97.62:46543 was elected leader, sending a full tablet report...
I20260929 15:45:29.145737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 8917
W20260929 15:45:29.163030 8708 connection.cc:535] server connection from 127.0.97.2:53035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:45:29.163785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9051
Sep 29 15:45:29 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:45:29.184739 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:29.165057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.0.97.3:36207)
0929 15:45:29.165713 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:29.165720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:29.165751 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:45:29.165815 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:29.165823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:29.166075 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:29.166280 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:29.167653 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:29.168792 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:29.170166 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:29.170740 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:29.170890 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:29.176651 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:29.176675 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:29.176697 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:29.176762 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:29.180601 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:29.181759 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:29.181770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:29.181773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:29.181962 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:29.182606 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:29.182616 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:29.182622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:29.183183 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:29.183577 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:29.183931 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 15:45:29.184324 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":246,"threads_started":1}
I20260929 15:45:29.186811 9174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:29.280928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9051
I20260929 15:45:29.299121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 8690
I20260929 15:45:29.301034 8752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:29.420493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 8690
I20260929 15:45:29.446458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46543
--webserver_interface=127.0.97.62
--webserver_port=34435
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:29.870033 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:29.870637 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:29.870898 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:29.881237 9197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:29.881507 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:29.881613 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:29.881734 9197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:29.881830 9197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:29.893859 9197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:29.895936 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:29.898108 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:29.912031 9202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:29.911998 9203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:29.911998 9205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:29.912926 9197 server_base.cc:1034] running on GCE node
I20260929 15:45:29.914656 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:29.917133 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:29.918493 9197 hybrid_clock.cc:648] HybridClock initialized: now 1790696729918385 us; error 97 us; skew 500 ppm
I20260929 15:45:29.918962 9197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:29 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:29.923087 9197 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:45:29.925411 9197 webserver.cc:460] Webserver started at http://127.0.97.62:34435/ using document root <none> and password file <none>
I20260929 15:45:29.926400 9197 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:29.926600 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:29.934753 9197 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:45:29.938946 9212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:29.941341 9197 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:45:29.941653 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "5ff86922ce154780aa16ae20ab1d1cd3"
format_stamp: "Formatted at 2026-09-29 15:45:26 on dist-test-slave-7491"
server_key: "816bd726d5f1fed1e8f9778db68ffc19"
server_key_iv: "b1b3c0f18af83a75047fc362f936d313"
server_key_version: "encryptionkey@0"
I20260929 15:45:29.943008 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:29.973651 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:29.980283 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:29.980640 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:30.003692 9197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46543
I20260929 15:45:30.003705 9263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46543 every 8 connection(s)
I20260929 15:45:30.005630 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:30.010514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9197
I20260929 15:45:30.011370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37183
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46543
--webserver_port=33577
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:30.015233 9264 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:45:30.020092 9264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Bootstrap starting.
I20260929 15:45:30.053397 9264 log.cc:824] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:30.068599 9264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:45:30.069362 9264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Bootstrap complete.
I20260929 15:45:30.086222 9264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:30.087585 9264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff86922ce154780aa16ae20ab1d1cd3, State: Initialized, Role: FOLLOWER
I20260929 15:45:30.088742 9264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:30.089129 9264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:30.089295 9264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:30.089753 9264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:45:30.094997 9264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:30.095716 9264 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff86922ce154780aa16ae20ab1d1cd3; no voters:
I20260929 15:45:30.096683 9264 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:45:30.096869 9268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:45:30.098225 9268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff86922ce154780aa16ae20ab1d1cd3, State: Running, Role: LEADER
I20260929 15:45:30.099249 9264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:30.099181 9268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } }
I20260929 15:45:30.103070 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff86922ce154780aa16ae20ab1d1cd3. Latest consensus state: current_term: 2 leader_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } } }
I20260929 15:45:30.103601 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:30.103096 9269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff86922ce154780aa16ae20ab1d1cd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46543 } } }
I20260929 15:45:30.103788 9269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:30.116413 9277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:30.123410 9277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:30.125546 9277 catalog_manager.cc:1259] Loaded cluster ID: 8c16549676284bf681b049f24e9284b7
I20260929 15:45:30.125723 9277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:30.129834 9277 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:30.130234 9285 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:45:30.131688 9277 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ff86922ce154780aa16ae20ab1d1cd3: Loaded TSK: 0
I20260929 15:45:30.133517 9277 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:45:30.464483 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:30.464896 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:30.465059 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:30.473918 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:30.474100 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:30.474366 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:30.488672 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33577
--tserver_master_addrs=127.0.97.62:46543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:30.490284 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:30.492254 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:30.511860 9290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:30.512104 9291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:30.513566 9293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:30.514431 9266 server_base.cc:1034] running on GCE node
I20260929 15:45:30.515241 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:30.516589 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:30.518018 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790696730517938 us; error 92 us; skew 500 ppm
I20260929 15:45:30.518476 9266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696730, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:30.522679 9266 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 15:45:30.524875 9266 webserver.cc:460] Webserver started at http://127.0.97.1:33577/ using document root <none> and password file <none>
I20260929 15:45:30.525738 9266 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:30.525904 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:30.532945 9266 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:30.537583 9300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:30.541316 9266 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:45:30.541682 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5140adfa69b84fa5bcfc8691674967b4"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "94c3e97a4535d376c28b3762c6ed37db"
server_key_iv: "bb16de39feb629c38cc60e073c1c50b3"
server_key_version: "encryptionkey@0"
I20260929 15:45:30.543720 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:30.579563 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:30.585994 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:30.586277 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:30.587383 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:30.589123 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:30.589255 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:30.589337 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:30.589396 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:30.632077 9266 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37183
I20260929 15:45:30.632143 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37183 every 8 connection(s)
I20260929 15:45:30.633849 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:30.636960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9266
I20260929 15:45:30.638062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45107
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46543
--webserver_port=44715
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:45:30.658257 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:30.638857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55401 (local address 127.0.97.62:46543)
0929 15:45:30.640014 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:30.640040 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:30.640676 (+ 636us) server_negotiation.cc:407] Connection header received
0929 15:45:30.643169 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:30.643182 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:30.643564 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:30.643970 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:30.645656 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.647047 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.649050 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.649839 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.650070 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:30.656774 (+ 6704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55401: BlockingRecv error: recv got EOF from 127.0.97.1:55401 (error 108)
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":297,"threads_started":1}
I20260929 15:45:30.658133 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:30.639109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:55401)
0929 15:45:30.640461 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:30.640513 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:30.642497 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:30.644128 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:30.644138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:30.644837 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:30.645402 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:30.645422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.647224 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.647234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:30.648891 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:30.648907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.650116 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.650124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:30.650396 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:30.651167 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:30.651191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:30.656135 (+ 4944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":768,"threads_started":1}
W20260929 15:45:30.659648 9413 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46543 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46543: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:30.670428 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:30.660710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:42161)
0929 15:45:30.660936 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:30.660959 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:30.661207 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:30.661717 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:30.661726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:30.662153 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:30.662342 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:30.662352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.663456 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.663469 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:30.664517 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:30.664531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.665581 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.665589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:30.665813 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:30.666499 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:30.666518 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:30.670102 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 15:45:30.670616 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:30.660891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42161 (local address 127.0.97.62:46543)
0929 15:45:30.661203 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:30.661211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:30.661244 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:45:30.661438 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:30.661450 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:30.661539 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:30.661664 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:30.662504 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.663305 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.664801 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.665323 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.665538 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:30.670362 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42161: BlockingRecv error: recv got EOF from 127.0.97.1:42161 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:30 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:30.682112 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:30.671372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:37879)
0929 15:45:30.671946 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:30.671962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:30.672130 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:30.672720 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:30.672726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:30.673216 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:30.673435 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:30.673445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.674590 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.674595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:30.675574 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:30.675587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.676449 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.676455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:30.676645 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:30.677335 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:30.677353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:30.681792 (+ 4439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456}
W20260929 15:45:30.682619 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:30.671461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37879 (local address 127.0.97.62:46543)
0929 15:45:30.671809 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:30.671819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:30.672051 (+ 232us) server_negotiation.cc:407] Connection header received
0929 15:45:30.672296 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:30.672305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:30.672492 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:30.672678 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:30.673620 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.674417 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.675740 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:30.676275 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:30.676423 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:30.682330 (+ 5907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37879: BlockingRecv error: recv got EOF from 127.0.97.1:37879 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260929 15:45:30.682796 9413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:45:31.041613 9417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:31.041939 9417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:31.042048 9417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:31.050222 9417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:31.050364 9417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:31.050482 9417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:31.061115 9417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44715
--tserver_master_addrs=127.0.97.62:46543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:31.063151 9417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:31.064652 9417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:31.080751 9422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:31.080777 9423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:31.080804 9425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:31.082027 9417 server_base.cc:1034] running on GCE node
I20260929 15:45:31.082806 9417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:31.084044 9417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:31.085373 9417 hybrid_clock.cc:648] HybridClock initialized: now 1790696731085325 us; error 47 us; skew 500 ppm
I20260929 15:45:31.085783 9417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:31.089577 9417 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 15:45:31.091460 9417 webserver.cc:460] Webserver started at http://127.0.97.2:44715/ using document root <none> and password file <none>
I20260929 15:45:31.092363 9417 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:31.092540 9417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:31.100510 9417 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:45:31.104519 9432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:31.106242 9417 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:31.106437 9417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "9860f269b34d4699a0df356ef3d75a39"
format_stamp: "Formatted at 2026-09-29 15:45:27 on dist-test-slave-7491"
server_key: "f9dbbcd88b92edb5274055869c4140d4"
server_key_iv: "03230255a5192266259f8544ba876fe8"
server_key_version: "encryptionkey@0"
I20260929 15:45:31.107487 9417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:31.128432 9417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:31.134344 9417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:31.134614 9417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:31.135671 9417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:31.137377 9417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:31.137483 9417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:31.137586 9417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:31.137647 9417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:31.177855 9417 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45107
I20260929 15:45:31.178049 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45107 every 8 connection(s)
I20260929 15:45:31.179559 9417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:31.180883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9417
I20260929 15:45:31.181569 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36207
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46543
--webserver_port=34187
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:45:31.202728 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.183177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44367 (local address 127.0.97.62:46543)
0929 15:45:31.183376 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.183384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.184965 (+ 1581us) server_negotiation.cc:407] Connection header received
0929 15:45:31.186995 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.187004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.187121 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.187272 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.189984 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.191285 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.193532 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.194384 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.194571 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.202275 (+ 7704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44367: BlockingRecv error: recv got EOF from 127.0.97.2:44367 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:45:31.203943 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.183763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:44367)
0929 15:45:31.184731 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.184809 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.186642 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.187779 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.187793 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.188724 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.189619 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.189662 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.191657 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.191664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.193281 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.193304 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.194769 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.194776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.195116 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.196134 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.196349 (+ 215us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.201936 (+ 5587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":439,"threads_started":1}
W20260929 15:45:31.204901 9545 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46543 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46543: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:31.216073 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.205757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:52073)
0929 15:45:31.205947 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.205971 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.206140 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.206544 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.206549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.206963 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.207134 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.207143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.208365 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.208371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.209367 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.209378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.210178 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.210184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.210384 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.211133 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.211153 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.215748 (+ 4595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:45:31.216523 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.205986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52073 (local address 127.0.97.62:46543)
0929 15:45:31.206220 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.206226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.206250 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:45:31.206329 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.206333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.206400 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.206510 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.207284 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.208150 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.209531 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.210013 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.210148 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.216089 (+ 5941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52073: BlockingRecv error: recv got EOF from 127.0.97.2:52073 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:31.227671 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.217142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:58913)
0929 15:45:31.217352 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.217391 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.217647 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.218156 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.218161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.218522 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.218744 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.218754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.220377 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.220383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.221456 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.221473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.222531 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.222537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.222723 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.223571 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.223593 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.227414 (+ 3821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:45:31.227967 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.217193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58913 (local address 127.0.97.62:46543)
0929 15:45:31.217369 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.217375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.217580 (+ 205us) server_negotiation.cc:407] Connection header received
0929 15:45:31.217793 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.217798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.217907 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.218047 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.218928 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.220073 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.221652 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.222289 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.222442 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.227727 (+ 5285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58913: BlockingRecv error: recv got EOF from 127.0.97.2:58913 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 15:45:31.228327 9545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:45:31.569759 9548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:31.570091 9548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:31.570189 9548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:31.578003 9548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:31.578138 9548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:31.578258 9548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:31.589017 9548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34187
--tserver_master_addrs=127.0.97.62:46543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:31.590471 9548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:31.591801 9548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:31.607486 9556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:31.607455 9554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:31.607565 9553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:31.613001 9548 server_base.cc:1034] running on GCE node
I20260929 15:45:31.613941 9548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:31.615198 9548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:31.616501 9548 hybrid_clock.cc:648] HybridClock initialized: now 1790696731616462 us; error 43 us; skew 500 ppm
I20260929 15:45:31.616897 9548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:31.627010 9548 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 15:45:31.629241 9548 webserver.cc:460] Webserver started at http://127.0.97.3:34187/ using document root <none> and password file <none>
I20260929 15:45:31.630126 9548 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:31.630312 9548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:31.638531 9548 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:45:31.642480 9563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:31.644768 9548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:45:31.645066 9548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "0c786009d113499bacbb17965098e3dd"
format_stamp: "Formatted at 2026-09-29 15:45:28 on dist-test-slave-7491"
server_key: "8f5b5217780cf83a1d3c766bd53da580"
server_key_iv: "e4967594c25ae2b0c34b61216a9df7a1"
server_key_version: "encryptionkey@0"
I20260929 15:45:31.646520 9548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:31.663761 9548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:31.669551 9548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:31.669927 9548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:31.671155 9548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:31.672832 9548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:31.672966 9548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:31.673128 9548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:31.673223 9548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:31.696110 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.683837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:37131)
0929 15:45:31.684575 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.684609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.684816 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.685365 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.685371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.686173 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.686503 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.686525 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.687876 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.687884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.689152 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.689167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.690519 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.690526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.690674 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.691452 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.691487 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.695738 (+ 4251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":434,"threads_started":1}
W20260929 15:45:31.696301 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.683905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37131 (local address 127.0.97.62:46543)
0929 15:45:31.684189 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.684198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.684755 (+ 557us) server_negotiation.cc:407] Connection header received
0929 15:45:31.684980 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.684985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.685104 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.685217 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.686747 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.687703 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.689403 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.690297 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.690466 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.695989 (+ 5523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37131: BlockingRecv error: recv got EOF from 127.0.97.1:37131 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:45:31.720211 9548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36207
I20260929 15:45:31.720263 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36207 every 8 connection(s)
I20260929 15:45:31.721777 9548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:31.731323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9548
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:45:31.745096 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.724777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36167 (local address 127.0.97.62:46543)
0929 15:45:31.725016 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.725023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.726650 (+ 1627us) server_negotiation.cc:407] Connection header received
0929 15:45:31.729358 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.729363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.729450 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.729580 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.731991 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.732980 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.736600 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.737421 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.737608 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.744807 (+ 7199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36167: BlockingRecv error: recv got EOF from 127.0.97.3:36167 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":86}
I20260929 15:45:31.746381 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.725222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:36167)
0929 15:45:31.726317 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.726384 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.728870 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.729987 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.729999 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.730804 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.731695 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.731723 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.733317 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.733327 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.736358 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.736384 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.737715 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.737722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.738025 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.738983 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.739018 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.744549 (+ 5531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":413,"threads_started":1}
W20260929 15:45:31.747617 9677 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46543 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:46543: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260929 15:45:31.754017 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.732497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.0.97.62:46543)
0929 15:45:31.733689 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.733694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.733712 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:45:31.733766 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.733769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.733836 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.733950 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.736600 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.737913 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.739856 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.740375 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.740481 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.743899 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:31.743916 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:31.743950 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:31.744026 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:31.748826 (+ 4800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:31.750110 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:31.750124 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:31.750139 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:31.750517 (+ 378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:31.751072 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:31.751078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:31.751082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:31.751760 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:31.752133 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:31.753037 (+ 904us) server_negotiation.cc:299] Negotiation successful
0929 15:45:31.753592 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":1060,"thread_start_us":873,"threads_started":1}
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:31.760074 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.748553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:47519)
0929 15:45:31.748736 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.748761 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.748905 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.749264 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.749268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.749793 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.749979 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.749989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.751290 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.751308 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.752581 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.752595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.753533 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.753538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.753736 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.754982 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.755017 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.759747 (+ 4730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 15:45:31.760474 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.748611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47519 (local address 127.0.97.62:46543)
0929 15:45:31.748905 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.748912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.748932 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:45:31.749024 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.749030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.749105 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.749232 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.750143 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.751146 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.752739 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.753361 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.753507 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.760094 (+ 6587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47519: BlockingRecv error: recv got EOF from 127.0.97.3:47519 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:31 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:31.769586 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:31.761015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:37117)
0929 15:45:31.761216 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:31.761237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:31.761368 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:31.761713 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:31.761717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:31.762071 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:31.762241 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.762251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.763303 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.763308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.764363 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:31.764375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.765135 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.765141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:31.765319 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.766041 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:31.766059 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:31.769232 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 15:45:31.769713 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:31.761094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37117 (local address 127.0.97.62:46543)
0929 15:45:31.761269 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:31.761278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:31.761298 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:45:31.761494 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:31.761499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:31.761578 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:31.761684 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:31.762391 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.763151 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.764511 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:31.764970 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:31.765116 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:31.769473 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37117: BlockingRecv error: recv got EOF from 127.0.97.3:37117 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 15:45:31.769987 9677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 15:45:32 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:32 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:32.238528 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:32.229265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:56767)
0929 15:45:32.229899 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:32.229921 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:32.230090 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:32.230470 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:32.230477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:32.230996 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:32.231204 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:32.231212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.232272 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.232278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:32.233259 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:32.233271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.234078 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.234084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:32.234273 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:32.235073 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:32.235093 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:32.238239 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":345,"threads_started":1}
W20260929 15:45:32.238722 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:32.229373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56767 (local address 127.0.97.62:46543)
0929 15:45:32.229601 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:32.229610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:32.230038 (+ 428us) server_negotiation.cc:407] Connection header received
0929 15:45:32.230255 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:32.230259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:32.230320 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:32.230429 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:32.231361 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.232078 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.233420 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.233876 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.234037 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:32.238462 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56767: BlockingRecv error: recv got EOF from 127.0.97.2:56767 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 15:45:32 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:32 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:32.707738 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:32.697679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:39341)
0929 15:45:32.698314 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:32.698332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:32.698528 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:32.698891 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:32.698896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:32.699324 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:32.699488 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:32.699497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.700798 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.700807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:32.701763 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:32.701777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.702676 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.702683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:32.702868 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:32.703700 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:32.703720 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:32.707456 (+ 3736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":389,"threads_started":1}
W20260929 15:45:32.708011 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:32.697784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39341 (local address 127.0.97.62:46543)
0929 15:45:32.698127 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:32.698135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:32.698435 (+ 300us) server_negotiation.cc:407] Connection header received
0929 15:45:32.698660 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:32.698666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:32.698740 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:32.698851 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:32.699642 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.700572 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.701926 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.702428 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.702665 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:32.707752 (+ 5087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39341: BlockingRecv error: recv got EOF from 127.0.97.1:39341 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 29 15:45:32 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:32 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:32.781235 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:32.770755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:37355)
0929 15:45:32.771412 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:32.771428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:32.771605 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:32.771953 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:32.771957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:32.772471 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:32.772677 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:32.772689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.773770 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.773776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:32.774757 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:32.774770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.776153 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.776159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:32.776365 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:32.777490 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:32.777517 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:32.780960 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":373,"threads_started":1}
W20260929 15:45:32.781706 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:32.770982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37355 (local address 127.0.97.62:46543)
0929 15:45:32.771271 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:32.771277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:32.771525 (+ 248us) server_negotiation.cc:407] Connection header received
0929 15:45:32.771737 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:32.771743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:32.771810 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:32.771919 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:32.772840 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.773601 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.774916 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:32.775878 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:32.776055 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:32.781275 (+ 5220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37355: BlockingRecv error: recv got EOF from 127.0.97.3:37355 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 29 15:45:33 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:33 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:33.249696 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:33.240062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:43811)
0929 15:45:33.240742 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:33.240760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:33.240935 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:33.241297 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:33.241302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:33.241740 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:33.241900 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:33.241909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.242957 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.242962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:33.243897 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:33.243938 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.244800 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.244805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:33.244987 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:33.245715 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:33.245740 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:33.249319 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":350,"threads_started":1}
W20260929 15:45:33.249774 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:33.240332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43811 (local address 127.0.97.62:46543)
0929 15:45:33.240593 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:33.240600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:33.240852 (+ 252us) server_negotiation.cc:407] Connection header received
0929 15:45:33.241081 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:33.241087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:33.241157 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:33.241264 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:33.242065 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.242796 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.244079 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.244629 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.244780 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:33.249555 (+ 4775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43811: BlockingRecv error: recv got EOF from 127.0.97.2:43811 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 15:45:33 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:33 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:33.719015 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:33.708975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:41217)
0929 15:45:33.709644 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:33.709661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:33.709839 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:33.710199 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:33.710204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:33.710607 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:33.710793 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:33.710802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.711975 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.711982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:33.713136 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:33.713151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.713983 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.713989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:33.714179 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:33.715089 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:33.715108 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:33.718566 (+ 3458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":327,"threads_started":1}
W20260929 15:45:33.719080 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:33.709281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41217 (local address 127.0.97.62:46543)
0929 15:45:33.709560 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:33.709567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:33.709754 (+ 187us) server_negotiation.cc:407] Connection header received
0929 15:45:33.709979 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:33.709991 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:33.710055 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:33.710169 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:33.710960 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.711804 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.713298 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.713817 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.713976 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:33.718809 (+ 4833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41217: BlockingRecv error: recv got EOF from 127.0.97.1:41217 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 29 15:45:33 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:33 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:33.792558 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:33.782541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:59867)
0929 15:45:33.783437 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:33.783498 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:33.783670 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:33.784286 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:33.784290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:33.784917 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:33.785185 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:33.785201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.786270 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.786275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:33.787218 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:33.787232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.788019 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.788024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:33.788310 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:33.788976 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:33.788995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:33.792288 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":403,"threads_started":1}
W20260929 15:45:33.792891 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:33.782693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59867 (local address 127.0.97.62:46543)
0929 15:45:33.783895 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:33.783903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:33.783922 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:45:33.783985 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:33.783990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:33.784072 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:33.784208 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:33.785397 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.786111 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.787396 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:33.787827 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:33.787941 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:33.792498 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59867: BlockingRecv error: recv got EOF from 127.0.97.3:59867 (error 108)
Metrics: {"server-negotiator.queue_time_us":1020}
Sep 29 15:45:34 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:34 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:34.261111 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:34.250845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:40525)
0929 15:45:34.251401 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:34.251418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:34.251556 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:34.251871 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:34.251875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:34.252345 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:34.252538 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:34.252548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.253665 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.253669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:34.254582 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:34.254595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.255604 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.255613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:34.255898 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:34.256929 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:34.256962 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:34.260786 (+ 3824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":262,"threads_started":1}
W20260929 15:45:34.261508 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:34.251022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40525 (local address 127.0.97.62:46543)
0929 15:45:34.251275 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:34.251281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:34.251506 (+ 225us) server_negotiation.cc:407] Connection header received
0929 15:45:34.251662 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:34.251667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:34.251732 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:34.251820 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:34.252728 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.253475 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.254749 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.255241 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.255407 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:34.261127 (+ 5720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40525: BlockingRecv error: recv got EOF from 127.0.97.2:40525 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 15:45:34 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:34 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:34.730612 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:34.720483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:46871)
0929 15:45:34.721387 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:34.721411 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:34.721629 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:34.721988 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:34.721993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:34.722414 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:34.722593 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:34.722601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.723645 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.723652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:34.724704 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:34.724717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.725474 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.725482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:34.725658 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:34.726796 (+ 1138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:34.726822 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:34.730360 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":671,"threads_started":1}
W20260929 15:45:34.730932 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:34.720609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46871 (local address 127.0.97.62:46543)
0929 15:45:34.720943 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:34.720954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:34.721547 (+ 593us) server_negotiation.cc:407] Connection header received
0929 15:45:34.721763 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:34.721776 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:34.721838 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:34.721957 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:34.722747 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.723504 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.724863 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.725290 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.725440 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:34.730657 (+ 5217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46871: BlockingRecv error: recv got EOF from 127.0.97.1:46871 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 29 15:45:34 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:34 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:34.804422 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:34.793868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:51727)
0929 15:45:34.794509 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:34.794557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:34.794870 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:34.795330 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:34.795335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:34.795801 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:34.796002 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:34.796011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.797598 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.797603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:34.798549 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:34.798569 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.799559 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.799564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:34.799703 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:34.800500 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:34.800518 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:34.804082 (+ 3564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
W20260929 15:45:34.804672 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:34.794081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51727 (local address 127.0.97.62:46543)
0929 15:45:34.794264 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:34.794268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:34.794744 (+ 476us) server_negotiation.cc:407] Connection header received
0929 15:45:34.795026 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:34.795033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:34.795131 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:34.795243 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:34.796215 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.797200 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.798767 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:34.799292 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:34.799469 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:34.804424 (+ 4955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51727: BlockingRecv error: recv got EOF from 127.0.97.3:51727 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 15:45:35 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:35 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:35.271618 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:35.262219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:40759)
0929 15:45:35.263056 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:35.263073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:35.263205 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:35.263563 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:35.263567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:35.263976 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:35.264128 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:35.264136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.265245 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.265253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:35.266248 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:35.266261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.267208 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.267214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:35.267441 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:35.268186 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:35.268205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:35.271352 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":327,"threads_started":1}
W20260929 15:45:35.271975 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:35.262484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40759 (local address 127.0.97.62:46543)
0929 15:45:35.262744 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:35.262759 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:35.263191 (+ 432us) server_negotiation.cc:407] Connection header received
0929 15:45:35.263323 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:35.263328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:35.263410 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:35.263502 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:35.264335 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.265098 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.266419 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.267050 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.267180 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:35.271663 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40759: BlockingRecv error: recv got EOF from 127.0.97.2:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 15:45:35 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:35 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:35.743373 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:35.731739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:50729)
0929 15:45:35.732496 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:35.732513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:35.732684 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:35.733055 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:35.733061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:35.733802 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:35.734078 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:35.734093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.735389 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.735396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:35.736610 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:35.736628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.737710 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.737739 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:35.738062 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:35.738984 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:35.739011 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:35.743054 (+ 4043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":448,"threads_started":1}
W20260929 15:45:35.743796 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:35.731957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50729 (local address 127.0.97.62:46543)
0929 15:45:35.732328 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:35.732335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:35.732611 (+ 276us) server_negotiation.cc:407] Connection header received
0929 15:45:35.732831 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:35.732836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:35.732911 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:35.733023 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:35.734417 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.735234 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.736884 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.737500 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.737659 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:35.743370 (+ 5711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50729: BlockingRecv error: recv got EOF from 127.0.97.1:50729 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Sep 29 15:45:35 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:35 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:35.819367 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:35.805614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:41651)
0929 15:45:35.808622 (+ 3008us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:35.808641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:35.808780 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:35.809154 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:35.809158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:35.809679 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:35.809836 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:35.809845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.811078 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.811086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:35.812374 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:35.812389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.813565 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.813571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:35.813772 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:35.814570 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:35.814591 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:35.818959 (+ 4368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2887,"thread_start_us":2767,"threads_started":1}
W20260929 15:45:35.819522 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:35.805722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41651 (local address 127.0.97.62:46543)
0929 15:45:35.806000 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:35.806028 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:35.808746 (+ 2718us) server_negotiation.cc:407] Connection header received
0929 15:45:35.808893 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:35.808897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:35.808968 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:35.809070 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:35.810022 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.810871 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.812651 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:35.813275 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:35.813447 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:35.819290 (+ 5843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41651: BlockingRecv error: recv got EOF from 127.0.97.3:41651 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:45:36 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:36 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:36.283674 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:36.272944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:47381)
0929 15:45:36.273541 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:36.273559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:36.273686 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:36.274128 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:36.274132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:36.274554 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:36.274952 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:36.274968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.276202 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.276208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:36.277172 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:36.277185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.278309 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.278314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:36.278520 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:36.279337 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:36.279355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:36.283380 (+ 4025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":308,"threads_started":1}
W20260929 15:45:36.283933 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:36.273234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47381 (local address 127.0.97.62:46543)
0929 15:45:36.273447 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:36.273451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:36.273646 (+ 195us) server_negotiation.cc:407] Connection header received
0929 15:45:36.273845 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:36.273850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:36.273924 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:36.274020 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:36.275208 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.275987 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.277348 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.278135 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.278275 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:36.283651 (+ 5376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47381: BlockingRecv error: recv got EOF from 127.0.97.2:47381 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 15:45:36 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:36 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:36.761333 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:36.744835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:51493)
0929 15:45:36.745529 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:36.745553 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:36.745712 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:36.746206 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:36.746209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:36.746666 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:36.746838 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:36.746879 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.748110 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.748114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:36.749245 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:36.749260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.750226 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.750232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:36.750573 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:36.757323 (+ 6750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:36.757352 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:36.761089 (+ 3737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":397,"threads_started":1}
W20260929 15:45:36.761785 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:36.744997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51493 (local address 127.0.97.62:46543)
0929 15:45:36.745218 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:36.745226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:36.745661 (+ 435us) server_negotiation.cc:407] Connection header received
0929 15:45:36.745964 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:36.745970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:36.746058 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:36.746145 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:36.747053 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.747809 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.749472 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.750004 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.750145 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:36.761510 (+ 11365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51493: BlockingRecv error: recv got EOF from 127.0.97.1:51493 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 15:45:36 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:36 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:36.831082 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:36.820837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:48853)
0929 15:45:36.821673 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:36.821694 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:36.821861 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:36.822415 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:36.822419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:36.822945 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:36.823380 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:36.823406 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.824941 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.824950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:36.826189 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:36.826203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.827004 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.827009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:36.827152 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:36.827870 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:36.827891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:36.830813 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":351,"threads_started":1}
W20260929 15:45:36.831422 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:36.821078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48853 (local address 127.0.97.62:46543)
0929 15:45:36.821469 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:36.821475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:36.821852 (+ 377us) server_negotiation.cc:407] Connection header received
0929 15:45:36.822133 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:36.822138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:36.822230 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:36.822346 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:36.823671 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.824778 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.826381 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:36.826820 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:36.827022 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:36.831087 (+ 4065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48853: BlockingRecv error: recv got EOF from 127.0.97.3:48853 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:45:37 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:37 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:37.294559 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:37.284989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:38815)
0929 15:45:37.285683 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:37.285700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:37.285878 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:37.286239 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:37.286244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:37.286674 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:37.286830 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:37.286839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.287848 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.287854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:37.289020 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:37.289034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.290125 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.290134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:37.290324 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:37.290970 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:37.290988 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:37.294313 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":410,"threads_started":1}
W20260929 15:45:37.295189 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:37.285194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38815 (local address 127.0.97.62:46543)
0929 15:45:37.285487 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:37.285496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:37.285803 (+ 307us) server_negotiation.cc:407] Connection header received
0929 15:45:37.286016 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:37.286021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:37.286095 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:37.286211 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:37.286987 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.287698 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.289265 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.289865 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.290006 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:37.294880 (+ 4874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38815: BlockingRecv error: recv got EOF from 127.0.97.2:38815 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 29 15:45:37 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:37 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:37.779753 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:37.762572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:43281)
0929 15:45:37.763252 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:37.763269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:37.763452 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:37.763804 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:37.763809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:37.764317 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:37.764487 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:37.764495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.765551 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.765558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:37.766469 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:37.766480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.767356 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.767362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:37.767525 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:37.773875 (+ 6350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:37.773909 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:37.779309 (+ 5400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":314,"threads_started":1}
W20260929 15:45:37.779840 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:37.762745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43281 (local address 127.0.97.62:46543)
0929 15:45:37.763058 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:37.763065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:37.763380 (+ 315us) server_negotiation.cc:407] Connection header received
0929 15:45:37.763590 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:37.763595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:37.763663 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:37.763772 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:37.764650 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.765393 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.766628 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.767188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.767330 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:37.779559 (+ 12229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43281: BlockingRecv error: recv got EOF from 127.0.97.1:43281 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 29 15:45:37 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:37 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:37.843544 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:37.832366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:49505)
0929 15:45:37.832962 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:37.832989 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:37.833128 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:37.833731 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:37.833736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:37.834232 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:37.834419 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:37.834428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.835794 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.835799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:37.836924 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:37.836938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.837958 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.837964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:37.838132 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:37.839219 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:37.839241 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:37.843274 (+ 4033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":347,"threads_started":1}
W20260929 15:45:37.843825 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:37.832637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49505 (local address 127.0.97.62:46543)
0929 15:45:37.832887 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:37.832892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:37.833122 (+ 230us) server_negotiation.cc:407] Connection header received
0929 15:45:37.833370 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:37.833375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:37.833467 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:37.833610 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:37.834666 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.835609 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.837131 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:37.837628 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:37.837847 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:37.843544 (+ 5697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49505: BlockingRecv error: recv got EOF from 127.0.97.3:49505 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:45:38 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:38 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:38.305428 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:38.295632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:41221)
0929 15:45:38.296294 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:38.296320 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:38.296445 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:38.296733 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:38.296736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:38.297161 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:38.297365 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:38.297375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.298594 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.298604 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:38.299954 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:38.299972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.301028 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.301033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:38.301159 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:38.301863 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:38.301882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:38.305146 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":261,"threads_started":1}
W20260929 15:45:38.305658 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:38.295737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41221 (local address 127.0.97.62:46543)
0929 15:45:38.296002 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:38.296009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:38.296400 (+ 391us) server_negotiation.cc:407] Connection header received
0929 15:45:38.296560 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:38.296564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:38.296632 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:38.296716 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:38.297535 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.298322 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.300248 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.300706 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.300823 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:38.305396 (+ 4573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41221: BlockingRecv error: recv got EOF from 127.0.97.2:41221 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 15:45:38 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:38 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:38.791056 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:38.781136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:52461)
0929 15:45:38.781847 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:38.781865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:38.781998 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:38.782488 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:38.782492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:38.783001 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:38.783359 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:38.783393 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.784677 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.784681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:38.785692 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:38.785706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.786534 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.786539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:38.786668 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:38.787414 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:38.787432 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:38.790802 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":421,"threads_started":1}
W20260929 15:45:38.791380 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:38.781172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52461 (local address 127.0.97.62:46543)
0929 15:45:38.781413 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:38.781422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:38.782071 (+ 649us) server_negotiation.cc:407] Connection header received
0929 15:45:38.782171 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:38.782175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:38.782260 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:38.782402 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:38.783612 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.784481 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.785873 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.786319 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.786433 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:38.791075 (+ 4642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52461: BlockingRecv error: recv got EOF from 127.0.97.1:52461 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 15:45:38 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:38 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:38.854854 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:38.844940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:41131)
0929 15:45:38.845648 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:38.845666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:38.845860 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:38.846330 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:38.846335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:38.846899 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:38.847179 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:38.847256 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.848626 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.848633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:38.849750 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:38.849765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.850586 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.850591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:38.850713 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:38.851466 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:38.851487 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:38.854579 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":300,"threads_started":1}
W20260929 15:45:38.855660 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:38.845263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41131 (local address 127.0.97.62:46543)
0929 15:45:38.845503 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:38.845509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:38.845835 (+ 326us) server_negotiation.cc:407] Connection header received
0929 15:45:38.846026 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:38.846034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:38.846116 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:38.846234 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:38.847410 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.848444 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.849945 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:38.850370 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:38.850483 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:38.855174 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41131: BlockingRecv error: recv got EOF from 127.0.97.3:41131 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 15:45:39 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:39 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:39.316461 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:39.306696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:48547)
0929 15:45:39.307230 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:39.307247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:39.307365 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:39.307646 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:39.307649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:39.308061 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:39.308253 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:39.308262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.309585 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.309589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:39.310538 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:39.310551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.311604 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.311615 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:39.311833 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:39.312996 (+ 1163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:39.313021 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:39.316071 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":308,"threads_started":1}
W20260929 15:45:39.316664 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:39.306876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48547 (local address 127.0.97.62:46543)
0929 15:45:39.307052 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:39.307057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:39.307362 (+ 305us) server_negotiation.cc:407] Connection header received
0929 15:45:39.307452 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:39.307456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:39.307520 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:39.307608 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:39.308409 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.309364 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.310750 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.311225 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.311348 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:39.316377 (+ 5029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48547: BlockingRecv error: recv got EOF from 127.0.97.2:48547 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 15:45:39 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:39 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:39.803421 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:39.792129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:60575)
0929 15:45:39.792748 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:39.792765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:39.792883 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:39.793212 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:39.793216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:39.793662 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:39.793840 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:39.793848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.795113 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.795120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:39.796387 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:39.796410 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.797447 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.797453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:39.797608 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:39.798414 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:39.798435 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:39.802916 (+ 4481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":332,"threads_started":1}
W20260929 15:45:39.803599 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:39.792463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60575 (local address 127.0.97.62:46543)
0929 15:45:39.792655 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:39.792659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:39.792874 (+ 215us) server_negotiation.cc:407] Connection header received
0929 15:45:39.792969 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:39.792973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:39.793035 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:39.793122 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:39.794029 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.794780 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.796609 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.797095 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.797420 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:39.803196 (+ 5776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60575: BlockingRecv error: recv got EOF from 127.0.97.1:60575 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 15:45:39 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:39 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:39.867534 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:39.856467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:52779)
0929 15:45:39.857536 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:39.857561 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:39.857885 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:39.858421 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:39.858429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:39.858946 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:39.859122 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:39.859133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.860293 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.860300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:39.861474 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:39.861488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.862593 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.862600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:39.862806 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:39.863642 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:39.863664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:39.867278 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":812,"threads_started":1}
W20260929 15:45:39.868105 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:39.856741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52779 (local address 127.0.97.62:46543)
0929 15:45:39.857047 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:39.857056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:39.857681 (+ 625us) server_negotiation.cc:407] Connection header received
0929 15:45:39.858077 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:39.858081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:39.858155 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:39.858265 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:39.859341 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.860078 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.861794 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:39.862352 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:39.862514 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:39.867900 (+ 5386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52779: BlockingRecv error: recv got EOF from 127.0.97.3:52779 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 15:45:40 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:40 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:40.329732 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:40.317730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:49525)
0929 15:45:40.318406 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:40.318424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:40.318581 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:40.319076 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:40.319081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:40.319765 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:40.320036 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:40.320047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.321728 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.321733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:40.323248 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:40.323271 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.324565 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.324577 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:40.324808 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:40.325545 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:40.325564 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:40.329342 (+ 3778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":407,"threads_started":1}
W20260929 15:45:40.329901 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:40.317844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49525 (local address 127.0.97.62:46543)
0929 15:45:40.318053 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:40.318058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:40.318602 (+ 544us) server_negotiation.cc:407] Connection header received
0929 15:45:40.318706 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:40.318711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:40.318794 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:40.318932 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:40.320357 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.321498 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.323504 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.324219 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.324360 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:40.329609 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49525: BlockingRecv error: recv got EOF from 127.0.97.2:49525 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:45:40 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:40 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:40.814425 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:40.804595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:35393)
0929 15:45:40.805148 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:40.805166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:40.805359 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:40.805727 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:40.805731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:40.806153 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:40.806309 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:40.806317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.807475 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.807481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:40.808552 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:40.808565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.809368 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.809374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:40.809549 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:40.810258 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:40.810278 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:40.814010 (+ 3732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":333,"threads_started":1}
W20260929 15:45:40.814528 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:40.804802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35393 (local address 127.0.97.62:46543)
0929 15:45:40.805042 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:40.805049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:40.805271 (+ 222us) server_negotiation.cc:407] Connection header received
0929 15:45:40.805496 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:40.805502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:40.805579 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:40.805695 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:40.806479 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.807311 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.808716 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.809189 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.809339 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:40.814258 (+ 4919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35393: BlockingRecv error: recv got EOF from 127.0.97.1:35393 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 15:45:40 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:40 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:40.880360 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:40.868775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:42531)
0929 15:45:40.869831 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:40.869858 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:40.870053 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:40.870758 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:40.870763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:40.871482 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:40.871674 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:40.871684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.872854 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.872859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:40.873851 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:40.873864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.874913 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.874920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:40.875108 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:40.875893 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:40.875918 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:40.879928 (+ 4010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":915,"thread_start_us":441,"threads_started":1}
W20260929 15:45:40.880530 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:40.869034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42531 (local address 127.0.97.62:46543)
0929 15:45:40.869371 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:40.869382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:40.869951 (+ 569us) server_negotiation.cc:407] Connection header received
0929 15:45:40.870408 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:40.870414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:40.870545 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:40.870672 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:40.871886 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.872688 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.874026 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:40.874613 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:40.874755 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:40.880245 (+ 5490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42531: BlockingRecv error: recv got EOF from 127.0.97.3:42531 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 29 15:45:41 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:41 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:41.342437 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:41.331143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:58733)
0929 15:45:41.331657 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:41.331676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:41.331811 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:41.332155 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:41.332158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:41.332599 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:41.332757 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:41.332766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.334942 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.334948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:41.336292 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:41.336309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.337544 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.337551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:41.337825 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:41.338603 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:41.338622 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:41.342194 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":298,"threads_started":1}
W20260929 15:45:41.342782 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:41.331322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58733 (local address 127.0.97.62:46543)
0929 15:45:41.331577 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:41.331584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:41.331791 (+ 207us) server_negotiation.cc:407] Connection header received
0929 15:45:41.331937 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:41.331942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:41.332017 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:41.332112 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:41.333087 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.334566 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.336455 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.337289 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.337424 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:41.342475 (+ 5051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58733: BlockingRecv error: recv got EOF from 127.0.97.2:58733 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 15:45:41 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:41 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:41.830716 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:41.816121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:33077)
0929 15:45:41.816829 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:41.816854 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:41.817030 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:41.817580 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:41.817585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:41.818206 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:41.818511 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:41.818527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.820098 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.820104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:41.821166 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:41.821183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.822418 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.822423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:41.822562 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:41.827176 (+ 4614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:41.827202 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:41.830378 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":358,"threads_started":1}
W20260929 15:45:41.831080 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:41.816491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33077 (local address 127.0.97.62:46543)
0929 15:45:41.816809 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:41.816815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:41.817034 (+ 219us) server_negotiation.cc:407] Connection header received
0929 15:45:41.817211 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:41.817219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:41.817336 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:41.817480 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:41.818756 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.819867 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.821390 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.822141 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.822294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:41.830753 (+ 8459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33077: BlockingRecv error: recv got EOF from 127.0.97.1:33077 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 29 15:45:41 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:41 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:41.892078 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:41.882039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:37073)
0929 15:45:41.882952 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:41.882973 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:41.883145 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:41.883885 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:41.883890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:41.884489 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:41.884717 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:41.884729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.886046 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.886050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:41.887004 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:41.887016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.887870 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.887875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:41.888021 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:41.888732 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:41.888752 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:41.891821 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":457,"threads_started":1}
W20260929 15:45:41.892374 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:41.882259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37073 (local address 127.0.97.62:46543)
0929 15:45:41.882586 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:41.882594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:41.883086 (+ 492us) server_negotiation.cc:407] Connection header received
0929 15:45:41.883470 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:41.883479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:41.883604 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:41.883776 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:41.884938 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.885855 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.887205 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:41.887659 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:41.887767 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:41.892069 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37073: BlockingRecv error: recv got EOF from 127.0.97.3:37073 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 29 15:45:42 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:42 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:42.353235 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:42.343793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:45743)
0929 15:45:42.344607 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:42.344624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:42.344793 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:42.345147 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:42.345152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:42.345595 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:42.345760 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:42.345768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.346811 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.346819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:42.347716 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:42.347727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.348780 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.348786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:42.348974 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:42.349673 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:42.349690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:42.352873 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":567,"threads_started":1}
W20260929 15:45:42.353361 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:42.343867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45743 (local address 127.0.97.62:46543)
0929 15:45:42.344122 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:42.344132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:42.344718 (+ 586us) server_negotiation.cc:407] Connection header received
0929 15:45:42.344925 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:42.344932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:42.345007 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:42.345119 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:42.345910 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.346670 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.347900 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.348595 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.348765 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:42.353100 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45743: BlockingRecv error: recv got EOF from 127.0.97.2:45743 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 15:45:42 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:42 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:42.842864 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:42.832120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:45163)
0929 15:45:42.832763 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:42.832779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:42.832965 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:42.833331 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:42.833336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:42.833799 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:42.833977 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:42.833985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.835025 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.835031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:42.836051 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:42.836065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.837094 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.837101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:42.837458 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:42.838447 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:42.838472 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:42.842453 (+ 3981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":362,"threads_started":1}
W20260929 15:45:42.843053 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:42.832322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45163 (local address 127.0.97.62:46543)
0929 15:45:42.832584 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:42.832593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:42.832891 (+ 298us) server_negotiation.cc:407] Connection header received
0929 15:45:42.833107 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:42.833118 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:42.833181 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:42.833295 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:42.834134 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.834874 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.836259 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.836687 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.837075 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:42.842798 (+ 5723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45163: BlockingRecv error: recv got EOF from 127.0.97.1:45163 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 15:45:42 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:42 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:42.905507 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:42.893378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:34465)
0929 15:45:42.893955 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:42.893973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:42.894157 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:42.895219 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:42.895224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:42.895735 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:42.895932 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:42.895942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.897112 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.897122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:42.898118 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:42.898132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.899308 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.899316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:42.899504 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:42.900709 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:42.900732 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:42.905237 (+ 4505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":333,"threads_started":1}
W20260929 15:45:42.905887 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:42.893455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34465 (local address 127.0.97.62:46543)
0929 15:45:42.893716 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:42.893725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:42.894080 (+ 355us) server_negotiation.cc:407] Connection header received
0929 15:45:42.894502 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:42.894507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:42.894578 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:42.895188 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:42.896111 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.896951 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.898381 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:42.898901 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:42.899108 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:42.905524 (+ 6416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34465: BlockingRecv error: recv got EOF from 127.0.97.3:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 15:45:43 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:43 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:43.365100 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:43.354690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:38281)
0929 15:45:43.355247 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:43.355267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:43.355421 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:43.355841 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:43.355847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:43.356485 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:43.356673 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:43.356682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.358216 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.358223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:43.359309 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:43.359329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.360158 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.360206 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:43.360392 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:43.361208 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:43.361231 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:43.364834 (+ 3603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":330,"threads_started":1}
W20260929 15:45:43.365543 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:43.354879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38281 (local address 127.0.97.62:46543)
0929 15:45:43.355257 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:43.355267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:43.355371 (+ 104us) server_negotiation.cc:407] Connection header received
0929 15:45:43.355588 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:43.355593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:43.355668 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:43.355815 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:43.356831 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.358005 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.359511 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.359971 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.360118 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:43.365135 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38281: BlockingRecv error: recv got EOF from 127.0.97.2:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Sep 29 15:45:43 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:43 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:43.854442 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:43.844282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:41159)
0929 15:45:43.844896 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:43.844923 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:43.845073 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:43.845461 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:43.845469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:43.845890 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:43.846058 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:43.846067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.847364 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.847368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:43.848680 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:43.848696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.849849 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.849854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:43.849988 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:43.850726 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:43.850746 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:43.854164 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":297,"threads_started":1}
W20260929 15:45:43.854962 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:43.844391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41159 (local address 127.0.97.62:46543)
0929 15:45:43.844599 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:43.844608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:43.845004 (+ 396us) server_negotiation.cc:407] Connection header received
0929 15:45:43.845216 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:43.845220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:43.845311 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:43.845413 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:43.846215 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.847156 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.848876 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.849609 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.849732 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:43.854638 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41159: BlockingRecv error: recv got EOF from 127.0.97.1:41159 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 15:45:43 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:43 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:43.916572 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:43.906927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:37035)
0929 15:45:43.907477 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:43.907494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:43.907668 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:43.908061 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:43.908066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:43.908500 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:43.908651 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:43.908659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.909689 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.909695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:43.910809 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:43.910822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.911805 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.911811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:43.912011 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:43.912772 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:43.912789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:43.916113 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":325,"threads_started":1}
W20260929 15:45:43.916808 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:43.907019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37035 (local address 127.0.97.62:46543)
0929 15:45:43.907247 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:43.907255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:43.907594 (+ 339us) server_negotiation.cc:407] Connection header received
0929 15:45:43.907803 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:43.907815 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:43.907906 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:43.908019 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:43.908801 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.909542 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.910966 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:43.911633 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:43.911776 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:43.916559 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37035: BlockingRecv error: recv got EOF from 127.0.97.3:37035 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 15:45:44 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:44 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:45:44.378058 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:44.366999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56471 (local address 127.0.97.62:46543)
0929 15:45:44.367208 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:44.367215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:44.367475 (+ 260us) server_negotiation.cc:407] Connection header received
0929 15:45:44.367679 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:44.367683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:44.367747 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:44.367836 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:44.369205 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.370105 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.371596 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.372072 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.372278 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:44.377811 (+ 5533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56471: BlockingRecv error: recv got EOF from 127.0.97.2:56471 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:45:44.377988 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:44.366689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:56471)
0929 15:45:44.367347 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:44.367372 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:44.367527 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:44.367926 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:44.367931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:44.368666 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:44.368889 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:44.368902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.370351 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.370355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:44.371373 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:44.371388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.372395 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.372401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:44.372541 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:44.373192 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:44.373211 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:44.377515 (+ 4304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":315,"threads_started":1}
Sep 29 15:45:44 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:44 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:44.867077 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:44.855855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:57243)
0929 15:45:44.856484 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:44.856501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:44.856694 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:44.857097 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:44.857100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:44.857655 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:44.857967 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:44.857997 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.859844 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.859849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:44.861272 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:44.861287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.862127 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.862131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:44.862295 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:44.863080 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:44.863099 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:44.866819 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":336,"threads_started":1}
W20260929 15:45:44.867446 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:44.856083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57243 (local address 127.0.97.62:46543)
0929 15:45:44.856370 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:44.856377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:44.856625 (+ 248us) server_negotiation.cc:407] Connection header received
0929 15:45:44.856838 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:44.856850 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:44.856916 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:44.857041 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:44.858269 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.859513 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.861475 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.861930 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.862067 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:44.867166 (+ 5099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57243: BlockingRecv error: recv got EOF from 127.0.97.1:57243 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 29 15:45:44 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:44 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:44.929313 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:44.918078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:39317)
0929 15:45:44.918874 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:44.918899 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:44.919222 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:44.919947 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:44.919956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:44.920697 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:44.920908 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:44.920918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.922376 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.922383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:44.923447 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:44.923464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.924686 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.924693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:44.924872 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:44.925590 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:44.925608 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:44.929043 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":462,"threads_started":1}
W20260929 15:45:44.929612 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:44.918387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39317 (local address 127.0.97.62:46543)
0929 15:45:44.918874 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:44.918881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:44.919019 (+ 138us) server_negotiation.cc:407] Connection header received
0929 15:45:44.919431 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:44.919435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:44.919613 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:44.919798 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:44.921215 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.922190 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.923682 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:44.924460 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:44.924625 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:44.929327 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39317: BlockingRecv error: recv got EOF from 127.0.97.3:39317 (error 108)
Metrics: {"server-negotiator.queue_time_us":315}
Sep 29 15:45:45 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:45 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:45.390158 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:45.379623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:57977)
0929 15:45:45.380392 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:45.380420 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:45.380560 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:45.381029 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:45.381035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:45.381545 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:45.381762 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:45.381772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.382980 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.382986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:45.384050 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:45.384072 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.384877 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.384883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:45.385077 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:45.385914 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:45.385934 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:45.389842 (+ 3908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":405,"threads_started":1}
W20260929 15:45:45.390447 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:45.379833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57977 (local address 127.0.97.62:46543)
0929 15:45:45.380131 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:45.380141 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:45.380511 (+ 370us) server_negotiation.cc:407] Connection header received
0929 15:45:45.380712 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:45.380717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:45.380824 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:45.380982 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:45.381925 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.382793 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.384238 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.384710 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.384855 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:45.390178 (+ 5323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57977: BlockingRecv error: recv got EOF from 127.0.97.2:57977 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 15:45:45 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:45 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:45.878245 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:45.868724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.1:45745)
0929 15:45:45.869431 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:45.869451 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:45.869645 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:45.870049 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:45.870054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:45.870539 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:45.870728 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:45.870736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.872040 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.872048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:45.873030 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:45.873042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.873797 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.873803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:45.873971 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:45.874889 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:45.874908 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:45.877869 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":409,"threads_started":1}
W20260929 15:45:45.878389 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:45.868880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45745 (local address 127.0.97.62:46543)
0929 15:45:45.869227 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:45.869234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:45.869560 (+ 326us) server_negotiation.cc:407] Connection header received
0929 15:45:45.869794 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:45.869806 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:45.869871 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:45.869990 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:45.870879 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.871881 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.873183 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.873604 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.873770 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:45.878134 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45745: BlockingRecv error: recv got EOF from 127.0.97.1:45745 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Sep 29 15:45:45 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:45 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:45.942889 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:45.930772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.3:34811)
0929 15:45:45.931393 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:45.931421 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:45.931573 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:45.932153 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:45.932240 (+ 87us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:45.932664 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:45.932876 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:45.932887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.935252 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.935259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:45.936462 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:45.936479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.937521 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.937529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:45.937676 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:45.938393 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:45.938418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:45.942559 (+ 4141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":349,"threads_started":1}
W20260929 15:45:45.943401 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:45.930955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34811 (local address 127.0.97.62:46543)
0929 15:45:45.931205 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:45.931210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:45.931598 (+ 388us) server_negotiation.cc:407] Connection header received
0929 15:45:45.931696 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:45.931700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:45.931806 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:45.931952 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:45.933106 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.934938 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.936627 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:45.937300 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:45.937458 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:45.942892 (+ 5434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34811: BlockingRecv error: recv got EOF from 127.0.97.3:34811 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 29 15:45:46 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:45:46 dist-test-slave-7491 krb5kdc[8677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:45:46.402608 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:46.391534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46543 (local address 127.0.97.2:45383)
0929 15:45:46.392123 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:46.392140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:46.392423 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:46.392761 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:46.392766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:46.393215 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:46.393413 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:46.393422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.394819 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.394825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:46.396055 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:46.396118 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.397045 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.397051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:46.397258 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:46.398107 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:46.398164 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:46.402128 (+ 3964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:46543: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":333,"threads_started":1}
W20260929 15:45:46.402742 9415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:46.391695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45383 (local address 127.0.97.62:46543)
0929 15:45:46.391949 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:46.391956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:46.392342 (+ 386us) server_negotiation.cc:407] Connection header received
0929 15:45:46.392549 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:46.392554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:46.392624 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:46.392734 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:46.393566 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.394535 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.396322 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.396853 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.397023 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:46.402364 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45383: BlockingRecv error: recv got EOF from 127.0.97.2:45383 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 15:45:46.735781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9266
I20260929 15:45:46.751839 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:46.737232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.97.1:37183)
0929 15:45:46.738128 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:46.738138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:46.738176 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:45:46.738354 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:46.738364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:46.738807 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:46.739047 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:46.740980 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.742317 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.743934 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.744773 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.744942 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:46.748151 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:46.748168 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:46.748192 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:46.748319 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:46.751159 (+ 2840us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:45:46.751499 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56800: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":526,"threads_started":1}
W20260929 15:45:46.751935 9726 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56800: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:45:46.752221 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9266: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:37183: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:45:46.752465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9266
I20260929 15:45:46.772411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9417
I20260929 15:45:46.788563 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:46.773674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.0.97.2:45107)
0929 15:45:46.774440 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:46.774448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:46.774481 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:45:46.774546 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:46.774552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:46.774781 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:46.775249 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:46.776836 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.778018 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.779845 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.780666 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.780815 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:46.784710 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:46.784721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:46.784744 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:46.784790 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:46.787903 (+ 3113us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:45:46.788291 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33038: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":408,"threads_started":1}
W20260929 15:45:46.788687 9727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33038: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:45:46.789073 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9417: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45107: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:45:46.789304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9417
I20260929 15:45:46.808223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9548
I20260929 15:45:46.823997 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:46.809499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.97.3:36207)
0929 15:45:46.810170 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:46.810177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:46.810206 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:45:46.810276 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:46.810283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:46.810518 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:46.810697 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:46.812042 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.813294 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.815133 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:46.815749 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:46.815895 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:46.819900 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:46.819923 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:46.819945 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:46.820004 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:46.823360 (+ 3356us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:45:46.823712 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39012: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":374,"threads_started":1}
W20260929 15:45:46.824321 9728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39012: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:45:46.824368 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9548: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36207: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:45:46.824537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9548
I20260929 15:45:46.842757 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9197
I20260929 15:45:46.844789 9259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:46.979104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9197
2026-09-29T15:45:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22289 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T15:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:47Z Disabled control of system clock
I20260929 15:45:47.071056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34183
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:45:47.470791 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:47.471135 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:47.471217 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:47.478816 9733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:47.478971 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:47.479015 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:47.479067 9733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:47.479107 9733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:47.488865 9733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34183
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:47.490422 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:47.492506 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:47.508571 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:47.508314 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:47.509255 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:47.509541 9733 server_base.cc:1034] running on GCE node
I20260929 15:45:47.510547 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:47.512467 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:47.513783 9733 hybrid_clock.cc:648] HybridClock initialized: now 1790696747513710 us; error 76 us; skew 500 ppm
I20260929 15:45:47.514225 9733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:47.524080 9733 webserver.cc:460] Webserver started at http://127.0.97.62:43549/ using document root <none> and password file <none>
I20260929 15:45:47.525218 9733 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:47.525388 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:47.525813 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:47.529687 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "248af35954bc4fc696b80389404b2912"
format_stamp: "Formatted at 2026-09-29 15:45:47 on dist-test-slave-7491"
server_key: "84835afc1fc9e9c30cd808db67a1b3c6"
server_key_iv: "2a2dcdd7fff6dd01e9b6c43743dbc021"
server_key_version: "encryptionkey@0"
I20260929 15:45:47.530858 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "248af35954bc4fc696b80389404b2912"
format_stamp: "Formatted at 2026-09-29 15:45:47 on dist-test-slave-7491"
server_key: "84835afc1fc9e9c30cd808db67a1b3c6"
server_key_iv: "2a2dcdd7fff6dd01e9b6c43743dbc021"
server_key_version: "encryptionkey@0"
I20260929 15:45:47.538350 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 15:45:47.543036 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:47.545264 9733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 15:45:47.545533 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "248af35954bc4fc696b80389404b2912"
format_stamp: "Formatted at 2026-09-29 15:45:47 on dist-test-slave-7491"
server_key: "84835afc1fc9e9c30cd808db67a1b3c6"
server_key_iv: "2a2dcdd7fff6dd01e9b6c43743dbc021"
server_key_version: "encryptionkey@0"
I20260929 15:45:47.545832 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:47.565733 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:47.566778 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:47.567128 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:47.589035 9733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34183
I20260929 15:45:47.589074 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34183 every 8 connection(s)
I20260929 15:45:47.590706 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:47.593933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9733
I20260929 15:45:47.594313 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:47.595259 388 external_mini_cluster.cc:1428] Setting key aea970d635e3c3e926f222f14d8b99ec
I20260929 15:45:47.596648 9799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:47.613847 9799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912: Bootstrap starting.
I20260929 15:45:47.615116 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:47.597986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.0.97.62:34183)
0929 15:45:47.599522 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:47.599538 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:47.599601 (+ 63us) server_negotiation.cc:407] Connection header received
0929 15:45:47.602949 (+ 3348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:47.602988 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:47.603674 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:47.604361 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:47.605287 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:47.606996 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:47.609243 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:47.610268 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:47.610485 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:47.612269 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:47.612304 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:47.612325 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:47.612979 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:47.613080 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:47.613107 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:47.613228 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:47.613568 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 15:45:47.614049 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":600,"threads_started":1}
I20260929 15:45:47.618019 9799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:47.619642 9799 log.cc:824] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:47.629588 9799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912: No bootstrap required, opened a new log
I20260929 15:45:47.637692 9799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248af35954bc4fc696b80389404b2912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34183 } }
I20260929 15:45:47.638161 9799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:47.638290 9799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248af35954bc4fc696b80389404b2912, State: Initialized, Role: FOLLOWER
I20260929 15:45:47.638947 9799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248af35954bc4fc696b80389404b2912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34183 } }
I20260929 15:45:47.639214 9799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:47.639396 9799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:47.639607 9799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:47.642787 9799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248af35954bc4fc696b80389404b2912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34183 } }
I20260929 15:45:47.643572 9799 leader_election.cc:302] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 248af35954bc4fc696b80389404b2912; no voters:
I20260929 15:45:47.644291 9799 leader_election.cc:288] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:47.644476 9804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:47.645924 9804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [term 1 LEADER]: Becoming Leader. State: Replica: 248af35954bc4fc696b80389404b2912, State: Running, Role: LEADER
I20260929 15:45:47.646574 9804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248af35954bc4fc696b80389404b2912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34183 } }
I20260929 15:45:47.646788 9799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:47.650291 9805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "248af35954bc4fc696b80389404b2912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248af35954bc4fc696b80389404b2912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34183 } } }
I20260929 15:45:47.650957 9805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:47.650132 9806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 248af35954bc4fc696b80389404b2912. Latest consensus state: current_term: 1 leader_uuid: "248af35954bc4fc696b80389404b2912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248af35954bc4fc696b80389404b2912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34183 } } }
I20260929 15:45:47.651732 9806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912 [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:47.654364 9813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:47.662254 9813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:47.672904 9813 catalog_manager.cc:1347] Generated new cluster ID: 9e997f6f33d745379d30c629d5837793
I20260929 15:45:47.673053 9813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:47.695411 9813 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:47.696969 9813 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:47.717675 9813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 248af35954bc4fc696b80389404b2912: Generated new TSK 0
I20260929 15:45:47.719460 9813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:45:47.737905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34183
--builtin_ntp_servers=127.0.97.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 15:45:48.178704 9823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:48.179103 9823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:48.179266 9823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:48.187391 9823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:48.187599 9823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:48.187885 9823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:48.199254 9823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:48.201473 9823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:48.204023 9823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:48.222419 9831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:48.222373 9828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:48.222729 9829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:48.224228 9823 server_base.cc:1034] running on GCE node
I20260929 15:45:48.224997 9823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:48.226312 9823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:48.227646 9823 hybrid_clock.cc:648] HybridClock initialized: now 1790696748227601 us; error 44 us; skew 500 ppm
I20260929 15:45:48.228060 9823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:48.231576 9823 webserver.cc:460] Webserver started at http://127.0.97.1:36977/ using document root <none> and password file <none>
I20260929 15:45:48.232906 9823 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:48.233228 9823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:48.233750 9823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:48.237848 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "0fce88e61cd24e1d9f0c01496a5cdd61"
format_stamp: "Formatted at 2026-09-29 15:45:48 on dist-test-slave-7491"
server_key: "5ea67123d1d2a00b0568b0e9ab5f1502"
server_key_iv: "c518c26380e7e79b24dd4c28aa3ecbfe"
server_key_version: "encryptionkey@0"
I20260929 15:45:48.238936 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0fce88e61cd24e1d9f0c01496a5cdd61"
format_stamp: "Formatted at 2026-09-29 15:45:48 on dist-test-slave-7491"
server_key: "5ea67123d1d2a00b0568b0e9ab5f1502"
server_key_iv: "c518c26380e7e79b24dd4c28aa3ecbfe"
server_key_version: "encryptionkey@0"
I20260929 15:45:48.245890 9823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:45:48.251049 9837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:48.253098 9823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:45:48.253285 9823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "0fce88e61cd24e1d9f0c01496a5cdd61"
format_stamp: "Formatted at 2026-09-29 15:45:48 on dist-test-slave-7491"
server_key: "5ea67123d1d2a00b0568b0e9ab5f1502"
server_key_iv: "c518c26380e7e79b24dd4c28aa3ecbfe"
server_key_version: "encryptionkey@0"
I20260929 15:45:48.253525 9823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:48.275888 9823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:48.277447 9823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:48.277773 9823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:48.279129 9823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:48.281108 9823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:48.281210 9823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:48.281301 9823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:48.281366 9823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:48.324556 9823 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45893
I20260929 15:45:48.324638 9949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45893 every 8 connection(s)
I20260929 15:45:48.326601 9823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:48.333396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9823
I20260929 15:45:48.333725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:48.334668 388 external_mini_cluster.cc:1428] Setting key 748c5b09fbf88a212f429ac381753f28
I20260929 15:45:48.338117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34183
--builtin_ntp_servers=127.0.97.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:45:48.343183 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:48.329676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51473 (local address 127.0.97.62:34183)
0929 15:45:48.330317 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:48.330323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:48.331963 (+ 1640us) server_negotiation.cc:407] Connection header received
0929 15:45:48.334087 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:48.334096 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:48.334217 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:48.334370 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:48.335890 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.336837 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.339173 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.339988 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.340218 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:48.342073 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:48.342083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:48.342092 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:48.342122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:48.342179 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:48.342198 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:48.342300 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:48.342595 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 15:45:48.342720 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":326,"threads_started":1}
I20260929 15:45:48.344376 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:48.330439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34183 (local address 127.0.97.1:51473)
0929 15:45:48.331696 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:48.331748 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:48.333670 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:48.334662 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:48.334672 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:48.334844 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:48.335676 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:48.335698 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.337120 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.337129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:48.338885 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:48.338911 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.340423 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.340431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:48.340739 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:48.341625 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:45:48.341642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:48.341695 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:45:48.341810 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:48.342338 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:48.342348 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:48.342356 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:45:48.342550 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 15:45:48.342965 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":405,"threads_started":1}
I20260929 15:45:48.346251 9950 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34183
I20260929 15:45:48.346679 9950 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:48.347764 9950 heartbeater.cc:511] Master 127.0.97.62:34183 requested a full tablet report, sending...
I20260929 15:45:48.350473 9764 ts_manager.cc:194] Registered new tserver with Master: 0fce88e61cd24e1d9f0c01496a5cdd61 (127.0.97.1:45893)
I20260929 15:45:48.352428 9764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51473
W20260929 15:45:48.725854 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:48.726240 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:48.726397 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:48.734127 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:48.734303 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:48.734479 9954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:48.744130 9954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:48.745638 9954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:48.747730 9954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:48.764549 9959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:48.764644 9962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:48.764593 9960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:48.770967 9954 server_base.cc:1034] running on GCE node
I20260929 15:45:48.771854 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:48.773042 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:48.774353 9954 hybrid_clock.cc:648] HybridClock initialized: now 1790696748774280 us; error 65 us; skew 500 ppm
I20260929 15:45:48.774763 9954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:48.777693 9954 webserver.cc:460] Webserver started at http://127.0.97.2:38207/ using document root <none> and password file <none>
I20260929 15:45:48.778506 9954 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:48.778611 9954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:48.778962 9954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:48.782263 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "b9614ac6ecd846b59b8de920cdd07272"
format_stamp: "Formatted at 2026-09-29 15:45:48 on dist-test-slave-7491"
server_key: "2030c67ab66313a93e9b373cb87b138a"
server_key_iv: "17857aaf5bbf075ba39b8385b2bd02d0"
server_key_version: "encryptionkey@0"
I20260929 15:45:48.783226 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b9614ac6ecd846b59b8de920cdd07272"
format_stamp: "Formatted at 2026-09-29 15:45:48 on dist-test-slave-7491"
server_key: "2030c67ab66313a93e9b373cb87b138a"
server_key_iv: "17857aaf5bbf075ba39b8385b2bd02d0"
server_key_version: "encryptionkey@0"
I20260929 15:45:48.790179 9954 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20260929 15:45:48.794718 9968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:48.796983 9954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:45:48.797232 9954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "b9614ac6ecd846b59b8de920cdd07272"
format_stamp: "Formatted at 2026-09-29 15:45:48 on dist-test-slave-7491"
server_key: "2030c67ab66313a93e9b373cb87b138a"
server_key_iv: "17857aaf5bbf075ba39b8385b2bd02d0"
server_key_version: "encryptionkey@0"
I20260929 15:45:48.797493 9954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:48.833253 9954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:48.834374 9954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:48.834734 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:48.835927 9954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:48.837922 9954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:48.838047 9954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:48.838179 9954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:48.838258 9954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:48.877058 9954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46035
I20260929 15:45:48.877130 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46035 every 8 connection(s)
I20260929 15:45:48.878718 9954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:48.882551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 9954
I20260929 15:45:48.882802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:48.883618 388 external_mini_cluster.cc:1428] Setting key 0a1aec509c49398314b11d16925139a0
I20260929 15:45:48.887292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34183
--builtin_ntp_servers=127.0.97.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:45:48.894377 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:48.882036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59943 (local address 127.0.97.62:34183)
0929 15:45:48.882693 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:48.882698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:48.884914 (+ 2216us) server_negotiation.cc:407] Connection header received
0929 15:45:48.887216 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:48.887222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:48.887322 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:48.887455 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:48.888768 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.889522 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.891233 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.891656 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.891804 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:48.893459 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:48.893464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:48.893468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:48.893490 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:48.893529 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:48.893541 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:48.893644 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:48.893898 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 15:45:48.894002 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":430,"threads_started":1}
I20260929 15:45:48.895632 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:48.883318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34183 (local address 127.0.97.2:59943)
0929 15:45:48.884719 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:48.884787 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:48.886897 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:48.887737 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:48.887746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:48.887854 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:48.888580 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:48.888599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.889697 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.889704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:48.891052 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:48.891066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:48.891915 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:48.891923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:48.892230 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:48.893075 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:45:48.893078 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:48.893123 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:45:48.893212 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:48.893644 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:48.893656 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:48.893667 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:45:48.893876 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 15:45:48.894344 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":409,"threads_started":1}
I20260929 15:45:48.897536 10081 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34183
I20260929 15:45:48.897878 10081 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:48.899214 10081 heartbeater.cc:511] Master 127.0.97.62:34183 requested a full tablet report, sending...
I20260929 15:45:48.901464 9764 ts_manager.cc:194] Registered new tserver with Master: b9614ac6ecd846b59b8de920cdd07272 (127.0.97.2:46035)
I20260929 15:45:48.903342 9764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59943
W20260929 15:45:49.294206 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:49.294546 10085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:49.294680 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:49.302968 10085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:49.303166 10085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:49.303381 10085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:49.314818 10085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:49.316509 10085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:49.318730 10085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:49.337247 10090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:49.337595 10091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:49.337849 10093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:49.342658 10085 server_base.cc:1034] running on GCE node
I20260929 15:45:49.343739 10085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:49.345347 10085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:49.346668 10085 hybrid_clock.cc:648] HybridClock initialized: now 1790696749346611 us; error 56 us; skew 500 ppm
I20260929 15:45:49.347234 10085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:49.350346 10085 webserver.cc:460] Webserver started at http://127.0.97.3:40209/ using document root <none> and password file <none>
I20260929 15:45:49.351392 10085 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:49.351509 10085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:49.351882 10085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:49.355391 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "12f07074c1f34b24acc74026c02eafe0"
format_stamp: "Formatted at 2026-09-29 15:45:49 on dist-test-slave-7491"
server_key: "d4abf088fc3fe1c8324f99b47d83dc82"
server_key_iv: "b318ce75c327158440b94f74515660ea"
server_key_version: "encryptionkey@0"
I20260929 15:45:49.356873 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "12f07074c1f34b24acc74026c02eafe0"
format_stamp: "Formatted at 2026-09-29 15:45:49 on dist-test-slave-7491"
server_key: "d4abf088fc3fe1c8324f99b47d83dc82"
server_key_iv: "b318ce75c327158440b94f74515660ea"
server_key_version: "encryptionkey@0"
I20260929 15:45:49.357079 9950 heartbeater.cc:503] Master 127.0.97.62:34183 was elected leader, sending a full tablet report...
I20260929 15:45:49.365768 10085 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260929 15:45:49.370354 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:49.372604 10085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:45:49.372814 10085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "12f07074c1f34b24acc74026c02eafe0"
format_stamp: "Formatted at 2026-09-29 15:45:49 on dist-test-slave-7491"
server_key: "d4abf088fc3fe1c8324f99b47d83dc82"
server_key_iv: "b318ce75c327158440b94f74515660ea"
server_key_version: "encryptionkey@0"
I20260929 15:45:49.373057 10085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:49.394542 10085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:49.395622 10085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:49.396005 10085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:49.397851 10085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:49.400102 10085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:49.400300 10085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:45:49.400609 10085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:49.400820 10085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:49.444999 10085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33687
I20260929 15:45:49.445050 10211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33687 every 8 connection(s)
I20260929 15:45:49.446943 10085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:49.450999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10085
I20260929 15:45:49.451218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:49.451761 388 external_mini_cluster.cc:1428] Setting key fe81daa2d615cbe21865b39e57a9f6a8
I20260929 15:45:49.463979 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:49.450568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48783 (local address 127.0.97.62:34183)
0929 15:45:49.451312 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:49.451320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:49.453150 (+ 1830us) server_negotiation.cc:407] Connection header received
0929 15:45:49.455160 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:49.455166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:49.455271 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:49.455417 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:49.457025 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.457856 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.459724 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.460548 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.460798 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:49.462966 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:49.462973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:49.462976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:49.463000 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:49.463046 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:49.463060 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:49.463175 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:49.463445 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 15:45:49.463571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":407,"threads_started":1}
I20260929 15:45:49.465528 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:49.451465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34183 (local address 127.0.97.3:48783)
0929 15:45:49.452936 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:49.452998 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:49.454843 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:49.455701 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:49.455711 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:49.455821 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:49.456732 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:49.456761 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.458056 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.458065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:49.459549 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:49.459569 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.461002 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.461015 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:49.461357 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:49.462461 (+ 1104us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:45:49.462465 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:49.462532 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:45:49.462644 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:49.463179 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:49.463193 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:49.463205 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:45:49.463437 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 15:45:49.463984 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":402,"threads_started":1}
I20260929 15:45:49.467851 10212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34183
I20260929 15:45:49.468466 10212 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:49.469708 10212 heartbeater.cc:511] Master 127.0.97.62:34183 requested a full tablet report, sending...
I20260929 15:45:49.471716 9763 ts_manager.cc:194] Registered new tserver with Master: 12f07074c1f34b24acc74026c02eafe0 (127.0.97.3:33687)
I20260929 15:45:49.472949 9763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:48783
I20260929 15:45:49.482882 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:45:49.491102 10214 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:49.489198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.97.62:34183)
0929 15:45:49.489668 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:49.489675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:49.490027 (+ 352us) server_negotiation.cc:407] Connection header received
0929 15:45:49.490277 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:49.490281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:49.490350 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:49.490468 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:49.490892 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50834: BlockingRecv error: recv got EOF from 127.0.0.1:50834 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
I20260929 15:45:49.492492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9823
I20260929 15:45:49.501916 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:49.493632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.0.97.1:45893)
0929 15:45:49.494223 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:49.494231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:49.494260 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:45:49.494324 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:49.494331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:49.494585 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:49.494783 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:49.495262 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.496428 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.497925 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.498541 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.498696 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:49.500086 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:49.500096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:49.500112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:49.500846 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:49.500897 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:49.500919 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:49.501059 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:49.501201 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 15:45:49.501484 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":277,"threads_started":1}
I20260929 15:45:49.503783 9945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:49.592248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9823
I20260929 15:45:49.613873 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9954
I20260929 15:45:49.623684 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:49.615005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.0.97.2:46035)
0929 15:45:49.615658 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:49.615667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:49.615704 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:45:49.615791 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:49.615800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:49.616125 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:49.616497 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:49.617032 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.618305 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.619790 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.620506 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.620676 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:49.622044 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:49.622052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:49.622068 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:49.622738 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:49.622792 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:49.622816 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:49.622982 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:49.623144 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 15:45:49.623299 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":325,"threads_started":1}
I20260929 15:45:49.625190 10076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:49.719070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9954
I20260929 15:45:49.736310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10085
I20260929 15:45:49.747905 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:49.737927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.0.97.3:33687)
0929 15:45:49.738862 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:49.738870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:49.738907 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:45:49.738977 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:49.738983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:49.739218 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:49.739465 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:49.739980 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.741332 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.743104 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:49.743853 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:49.744029 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:49.745857 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:49.745866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:49.745882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:49.746525 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:49.746578 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:49.746600 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:49.746758 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:49.746935 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 15:45:49.747405 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":586,"threads_started":1}
I20260929 15:45:49.749469 10207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:49.843269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10085
I20260929 15:45:49.861342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 9733
I20260929 15:45:49.862634 9794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:49.974471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 9733
2026-09-29T15:45:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2993 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T15:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:50Z Disabled control of system clock
I20260929 15:45:50.063212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38859
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:45:50.456945 10232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:50.457345 10232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:50.457508 10232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:50.465634 10232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:50.465821 10232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:50.465924 10232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:50.466032 10232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:50.466120 10232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:50.477149 10232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:50.478993 10232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:50.481555 10232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:50.497807 10237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:50.497678 10240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:50.497769 10238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:50.498673 10232 server_base.cc:1034] running on GCE node
I20260929 15:45:50.499627 10232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:50.501485 10232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:50.502852 10232 hybrid_clock.cc:648] HybridClock initialized: now 1790696750502786 us; error 58 us; skew 500 ppm
I20260929 15:45:50.503309 10232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:50.506057 10232 webserver.cc:460] Webserver started at http://127.0.97.62:32803/ using document root <none> and password file <none>
I20260929 15:45:50.506898 10232 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:50.507048 10232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:50.507404 10232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:50.510751 10232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "d770f7e02d8b406b8cbf65e87cfdf92a"
format_stamp: "Formatted at 2026-09-29 15:45:50 on dist-test-slave-7491"
server_key: "447e62fd04e841098bf240b25ffeef21"
server_key_iv: "56170ea2a541620409bd89653eb5d2d8"
server_key_version: "encryptionkey@0"
I20260929 15:45:50.511646 10232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "d770f7e02d8b406b8cbf65e87cfdf92a"
format_stamp: "Formatted at 2026-09-29 15:45:50 on dist-test-slave-7491"
server_key: "447e62fd04e841098bf240b25ffeef21"
server_key_iv: "56170ea2a541620409bd89653eb5d2d8"
server_key_version: "encryptionkey@0"
I20260929 15:45:50.518066 10232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 15:45:50.522938 10246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:50.525354 10232 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:45:50.525642 10232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "d770f7e02d8b406b8cbf65e87cfdf92a"
format_stamp: "Formatted at 2026-09-29 15:45:50 on dist-test-slave-7491"
server_key: "447e62fd04e841098bf240b25ffeef21"
server_key_iv: "56170ea2a541620409bd89653eb5d2d8"
server_key_version: "encryptionkey@0"
I20260929 15:45:50.525938 10232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:50.534034 10232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:50.535262 10232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:50.535642 10232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:50.560974 10232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38859
I20260929 15:45:50.561103 10297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38859 every 8 connection(s)
I20260929 15:45:50.562693 10232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:50.567107 10298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:50.571494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10232
I20260929 15:45:50.571731 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:50.572420 388 external_mini_cluster.cc:1428] Setting key 6e5448d72ec26b23a1d86a9875d4c50b
I20260929 15:45:50.579906 10298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a: Bootstrap starting.
I20260929 15:45:50.582016 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:50.575089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.0.97.62:38859)
0929 15:45:50.576283 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:50.576295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:50.576352 (+ 57us) server_negotiation.cc:407] Connection header received
0929 15:45:50.578119 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:50.578144 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:50.578639 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:50.579011 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:50.580314 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:50.580327 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:50.580340 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:50.580356 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:50.580425 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:50.580443 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:50.580636 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:50.580733 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 15:45:50.581029 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":425,"threads_started":1}
I20260929 15:45:50.583575 10298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:50.584851 10298 log.cc:824] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:50.587931 10298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a: No bootstrap required, opened a new log
I20260929 15:45:50.596448 10298 raft_consensus.cc:348] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38859 } }
I20260929 15:45:50.597048 10298 raft_consensus.cc:374] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:50.597172 10298 raft_consensus.cc:729] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d770f7e02d8b406b8cbf65e87cfdf92a, State: Initialized, Role: FOLLOWER
I20260929 15:45:50.597990 10298 consensus_queue.cc:260] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38859 } }
I20260929 15:45:50.598294 10298 raft_consensus.cc:388] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:50.598454 10298 raft_consensus.cc:482] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:50.598660 10298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:50.601665 10298 raft_consensus.cc:504] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38859 } }
I20260929 15:45:50.602569 10298 leader_election.cc:302] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d770f7e02d8b406b8cbf65e87cfdf92a; no voters:
I20260929 15:45:50.603606 10298 leader_election.cc:288] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:50.603751 10303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:50.604879 10303 raft_consensus.cc:686] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [term 1 LEADER]: Becoming Leader. State: Replica: d770f7e02d8b406b8cbf65e87cfdf92a, State: Running, Role: LEADER
I20260929 15:45:50.605474 10303 consensus_queue.cc:237] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38859 } }
I20260929 15:45:50.606417 10298 sys_catalog.cc:564] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:50.608740 10304 sys_catalog.cc:455] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38859 } } }
I20260929 15:45:50.609675 10304 sys_catalog.cc:458] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:50.608741 10305 sys_catalog.cc:455] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d770f7e02d8b406b8cbf65e87cfdf92a. Latest consensus state: current_term: 1 leader_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d770f7e02d8b406b8cbf65e87cfdf92a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38859 } } }
I20260929 15:45:50.610198 10305 sys_catalog.cc:458] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:50.613893 10310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:50.620302 10310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:50.632685 10310 catalog_manager.cc:1347] Generated new cluster ID: 7d34c95e10ae45b984a55ea79195bc11
I20260929 15:45:50.632839 10310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:50.650988 10310 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:50.651530 10310 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:50.660542 10310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d770f7e02d8b406b8cbf65e87cfdf92a: Generated new TSK 0
I20260929 15:45:50.661561 10310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:45:50.673633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38859
--builtin_ntp_servers=127.0.97.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:45:51.069123 10322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:51.069545 10322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:51.069880 10322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:51.078059 10322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:51.078202 10322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:51.078402 10322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:51.089761 10322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:51.091297 10322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:51.093405 10322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:51.110607 10328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:51.110589 10327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:51.110680 10330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:51.111644 10322 server_base.cc:1034] running on GCE node
I20260929 15:45:51.112505 10322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:51.113873 10322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:51.115142 10322 hybrid_clock.cc:648] HybridClock initialized: now 1790696751115078 us; error 81 us; skew 500 ppm
I20260929 15:45:51.115530 10322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:51.119175 10322 webserver.cc:460] Webserver started at http://127.0.97.1:40835/ using document root <none> and password file <none>
I20260929 15:45:51.120302 10322 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:51.120471 10322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:51.120867 10322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:51.125191 10322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "888a6bf64d7b464cb73381895f5046cb"
format_stamp: "Formatted at 2026-09-29 15:45:51 on dist-test-slave-7491"
server_key: "082ca8ed723e68294e902115646fd278"
server_key_iv: "3ad6d787eb6c1dbefca6463b3567c552"
server_key_version: "encryptionkey@0"
I20260929 15:45:51.126588 10322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "888a6bf64d7b464cb73381895f5046cb"
format_stamp: "Formatted at 2026-09-29 15:45:51 on dist-test-slave-7491"
server_key: "082ca8ed723e68294e902115646fd278"
server_key_iv: "3ad6d787eb6c1dbefca6463b3567c552"
server_key_version: "encryptionkey@0"
I20260929 15:45:51.133486 10322 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 15:45:51.138021 10336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:51.140555 10322 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260929 15:45:51.140863 10322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "888a6bf64d7b464cb73381895f5046cb"
format_stamp: "Formatted at 2026-09-29 15:45:51 on dist-test-slave-7491"
server_key: "082ca8ed723e68294e902115646fd278"
server_key_iv: "3ad6d787eb6c1dbefca6463b3567c552"
server_key_version: "encryptionkey@0"
I20260929 15:45:51.141232 10322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:51.149063 10322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:51.150058 10322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:51.150386 10322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:51.151503 10322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:51.153156 10322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:51.153302 10322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:51.153438 10322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:51.153524 10322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:51.195111 10322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33837
I20260929 15:45:51.195155 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33837 every 8 connection(s)
I20260929 15:45:51.197057 10322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:51.202749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10322
I20260929 15:45:51.202967 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:51.203787 388 external_mini_cluster.cc:1428] Setting key 220682c75814420364ba0b3f4e45f852
I20260929 15:45:51.208485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38859
--builtin_ntp_servers=127.0.97.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:45:51.209241 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:51.200806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34773 (local address 127.0.97.62:38859)
0929 15:45:51.201435 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:51.201441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:51.203129 (+ 1688us) server_negotiation.cc:407] Connection header received
0929 15:45:51.205645 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:51.205650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:51.205716 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:51.205852 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:51.208009 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:51.208014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:51.208018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:51.208045 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:51.208097 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:51.208111 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:51.208318 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:51.208710 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 15:45:51.208823 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":308,"threads_started":1}
I20260929 15:45:51.210664 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:51.201504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38859 (local address 127.0.97.1:34773)
0929 15:45:51.202898 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:51.202960 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:51.205172 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:51.206211 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:51.206224 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:51.206380 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:51.207478 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:45:51.207483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:51.207547 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:45:51.207692 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:51.208318 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:51.208332 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:51.208342 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:45:51.208708 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 15:45:51.209093 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":361,"threads_started":1}
I20260929 15:45:51.212918 10449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38859
I20260929 15:45:51.213335 10449 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:51.214160 10449 heartbeater.cc:511] Master 127.0.97.62:38859 requested a full tablet report, sending...
I20260929 15:45:51.217242 10263 ts_manager.cc:194] Registered new tserver with Master: 888a6bf64d7b464cb73381895f5046cb (127.0.97.1:33837)
W20260929 15:45:51.610654 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:51.610998 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:51.611089 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:51.619633 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:51.619791 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:51.620041 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:51.631528 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:51.633193 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:51.636365 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:51.653903 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:51.654055 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:51.654054 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:51.655260 10453 server_base.cc:1034] running on GCE node
I20260929 15:45:51.656047 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:51.657256 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:51.658614 10453 hybrid_clock.cc:648] HybridClock initialized: now 1790696751658522 us; error 53 us; skew 500 ppm
I20260929 15:45:51.659195 10453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:51.662810 10453 webserver.cc:460] Webserver started at http://127.0.97.2:35659/ using document root <none> and password file <none>
I20260929 15:45:51.663810 10453 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:51.663931 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:51.664433 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:51.668319 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "ac00a329a5fa4429ba0a46bb24bcde74"
format_stamp: "Formatted at 2026-09-29 15:45:51 on dist-test-slave-7491"
server_key: "e47c76df6e83ff618ceba5130abe1f65"
server_key_iv: "3ffc375ece5c90498c2997fb1daa788f"
server_key_version: "encryptionkey@0"
I20260929 15:45:51.669314 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ac00a329a5fa4429ba0a46bb24bcde74"
format_stamp: "Formatted at 2026-09-29 15:45:51 on dist-test-slave-7491"
server_key: "e47c76df6e83ff618ceba5130abe1f65"
server_key_iv: "3ffc375ece5c90498c2997fb1daa788f"
server_key_version: "encryptionkey@0"
I20260929 15:45:51.678133 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 15:45:51.683665 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:51.686012 10453 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:45:51.686380 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "ac00a329a5fa4429ba0a46bb24bcde74"
format_stamp: "Formatted at 2026-09-29 15:45:51 on dist-test-slave-7491"
server_key: "e47c76df6e83ff618ceba5130abe1f65"
server_key_iv: "3ffc375ece5c90498c2997fb1daa788f"
server_key_version: "encryptionkey@0"
I20260929 15:45:51.686695 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:51.693697 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:51.694976 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:51.695353 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:51.696734 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:51.698767 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:51.698868 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:51.698997 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:51.699081 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:51.741845 10453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43635
I20260929 15:45:51.741916 10579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43635 every 8 connection(s)
I20260929 15:45:51.743629 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:51.750746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10453
I20260929 15:45:51.750982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:51.751914 388 external_mini_cluster.cc:1428] Setting key ce565cf544a9d54ba6c18f392094354f
I20260929 15:45:51.753726 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:51.747130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45901 (local address 127.0.97.62:38859)
0929 15:45:51.747738 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:51.747743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:51.749129 (+ 1386us) server_negotiation.cc:407] Connection header received
0929 15:45:51.751048 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:51.751055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:51.751166 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:51.751303 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:51.752885 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:51.752891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:51.752896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:51.752924 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:51.752965 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:51.752978 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:51.753132 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:51.753317 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 15:45:51.753390 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":351,"threads_started":1}
I20260929 15:45:51.754781 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:51.747859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38859 (local address 127.0.97.2:45901)
0929 15:45:51.748943 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:51.748990 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:51.750679 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:51.751588 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:51.751598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:51.751704 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:51.752535 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:45:51.752538 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:51.752580 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:45:51.752674 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:51.753077 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:51.753087 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:51.753097 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:45:51.753422 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 15:45:51.753722 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":383,"threads_started":1}
I20260929 15:45:51.755110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38859
--builtin_ntp_servers=127.0.97.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:45:51.757072 10580 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38859
I20260929 15:45:51.757516 10580 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:51.758311 10580 heartbeater.cc:511] Master 127.0.97.62:38859 requested a full tablet report, sending...
I20260929 15:45:51.760236 10263 ts_manager.cc:194] Registered new tserver with Master: ac00a329a5fa4429ba0a46bb24bcde74 (127.0.97.2:43635)
W20260929 15:45:52.172655 10584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:52.173043 10584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:52.173153 10584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:52.182619 10584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:52.182786 10584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:52.182917 10584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:52.196365 10584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:52.198184 10584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:52.200806 10584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:52.217267 10589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:52.217382 10590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:52.217422 10592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:52.218852 10584 server_base.cc:1034] running on GCE node
I20260929 15:45:52.219936 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:52.221580 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:52.222239 10449 heartbeater.cc:503] Master 127.0.97.62:38859 was elected leader, sending a full tablet report...
I20260929 15:45:52.223394 10584 hybrid_clock.cc:648] HybridClock initialized: now 1790696752223264 us; error 91 us; skew 500 ppm
I20260929 15:45:52.223912 10584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:45:52.227206 10584 webserver.cc:460] Webserver started at http://127.0.97.3:33733/ using document root <none> and password file <none>
I20260929 15:45:52.228094 10584 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:52.228284 10584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:52.228690 10584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:52.232314 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "b7730f815cbe40109e19dda98c753b22"
format_stamp: "Formatted at 2026-09-29 15:45:52 on dist-test-slave-7491"
server_key: "11d4e74c626aead3df23696d1bf5dcc9"
server_key_iv: "6492a9f58c12d2f3e326aa821c8adab0"
server_key_version: "encryptionkey@0"
I20260929 15:45:52.233309 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b7730f815cbe40109e19dda98c753b22"
format_stamp: "Formatted at 2026-09-29 15:45:52 on dist-test-slave-7491"
server_key: "11d4e74c626aead3df23696d1bf5dcc9"
server_key_iv: "6492a9f58c12d2f3e326aa821c8adab0"
server_key_version: "encryptionkey@0"
I20260929 15:45:52.240538 10584 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260929 15:45:52.245682 10598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:52.247929 10584 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:45:52.248251 10584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "b7730f815cbe40109e19dda98c753b22"
format_stamp: "Formatted at 2026-09-29 15:45:52 on dist-test-slave-7491"
server_key: "11d4e74c626aead3df23696d1bf5dcc9"
server_key_iv: "6492a9f58c12d2f3e326aa821c8adab0"
server_key_version: "encryptionkey@0"
I20260929 15:45:52.248684 10584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:52.256068 10584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:52.257225 10584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:52.257555 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:52.258777 10584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:52.260751 10584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:52.260903 10584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:52.261046 10584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:52.261138 10584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:52.300328 10584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34461
I20260929 15:45:52.300559 10710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34461 every 8 connection(s)
I20260929 15:45:52.301929 10584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:52.308108 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10584
I20260929 15:45:52.308320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:52.308807 388 external_mini_cluster.cc:1428] Setting key 3bfecd664840c0f9f509434731dff6e3
I20260929 15:45:52.313969 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:52.305864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37407 (local address 127.0.97.62:38859)
0929 15:45:52.306404 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:52.306409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:52.308488 (+ 2079us) server_negotiation.cc:407] Connection header received
0929 15:45:52.310891 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:52.310898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:52.310979 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:52.311126 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:52.313095 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:52.313100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:52.313103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:52.313121 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:52.313166 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:52.313179 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:52.313291 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:52.313531 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 15:45:52.313593 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":303,"threads_started":1}
I20260929 15:45:52.315217 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:52.306665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38859 (local address 127.0.97.3:37407)
0929 15:45:52.308217 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:52.308274 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:52.310538 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:52.311615 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:52.311629 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:52.311785 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:52.312722 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:45:52.312725 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:52.312777 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:45:52.312895 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:52.313280 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:52.313288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:52.313295 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:45:52.313561 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 15:45:52.313901 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":476,"threads_started":1}
I20260929 15:45:52.317317 10711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38859
I20260929 15:45:52.317744 10711 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:52.318585 10711 heartbeater.cc:511] Master 127.0.97.62:38859 requested a full tablet report, sending...
I20260929 15:45:52.320906 10263 ts_manager.cc:194] Registered new tserver with Master: b7730f815cbe40109e19dda98c753b22 (127.0.97.3:34461)
I20260929 15:45:52.326782 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:45:52.334779 10713 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:45:52.332655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.0.97.62:38859)
0929 15:45:52.332870 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:52.332875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:52.333399 (+ 524us) server_negotiation.cc:407] Connection header received
0929 15:45:52.333704 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:52.333710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:52.333800 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:52.334108 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:52.334561 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38370: BlockingRecv error: recv got EOF from 127.0.0.1:38370 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:45:52.336228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10322
I20260929 15:45:52.341456 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:52.337731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.0.97.1:33837)
0929 15:45:52.338480 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:52.338489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:52.338521 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:45:52.338597 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:52.338605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:52.338893 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:52.339118 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:52.340529 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:52.340538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:52.340552 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:52.340564 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:52.340612 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:52.340627 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:52.340872 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:52.340961 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 15:45:52.341100 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":321,"threads_started":1}
I20260929 15:45:52.343149 10444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:52.436532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10322
I20260929 15:45:52.456277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10453
I20260929 15:45:52.461490 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:52.457628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.0.97.2:43635)
0929 15:45:52.458311 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:52.458323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:52.458360 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:45:52.458454 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:52.458463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:52.458788 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:52.458976 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:52.460445 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:52.460456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:52.460473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:52.460511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:52.460585 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:52.460605 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:52.460754 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:52.461029 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 15:45:52.461152 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":347,"threads_started":1}
I20260929 15:45:52.463235 10575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:52.551661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10453
I20260929 15:45:52.570546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10584
I20260929 15:45:52.579855 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:52.571887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.0.97.3:34461)
0929 15:45:52.572646 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:52.572663 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:52.572695 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:45:52.572761 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:52.572769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:52.573012 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:52.573226 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:52.578806 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:52.578819 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:52.578839 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:45:52.578878 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:52.578972 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:45:52.578997 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:52.579244 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:52.579337 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 15:45:52.579486 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":431,"threads_started":1}
I20260929 15:45:52.581790 10706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:52.672585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10584
I20260929 15:45:52.691665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10232
I20260929 15:45:52.693147 10293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:52.805601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10232
2026-09-29T15:45:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2826 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:45:52 dist-test-slave-7491 krb5kdc[10730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:45:52 dist-test-slave-7491 krb5kdc[10730](info): set up 2 sockets
Sep 29 15:45:52 dist-test-slave-7491 krb5kdc[10730](info): commencing operation
krb5kdc: starting...
W20260929 15:45:53.965272 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:45:54 dist-test-slave-7491 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:45:54.241272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33325
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:54.634127 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:54.634549 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:54.634737 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:54.643000 10743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:45:54.643198 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:54.643326 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:54.643457 10743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:45:54.643541 10743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:45:54.655236 10743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:54.657305 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:54.659341 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:54.675680 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:54.675772 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:54.675668 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:54.679049 10743 server_base.cc:1034] running on GCE node
I20260929 15:45:54.680066 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:54.681931 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:54.683295 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790696754683234 us; error 52 us; skew 500 ppm
I20260929 15:45:54.683781 10743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:54 dist-test-slave-7491 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:54.687588 10743 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:45:54.689735 10743 webserver.cc:460] Webserver started at http://127.0.97.62:44963/ using document root <none> and password file <none>
I20260929 15:45:54.690598 10743 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:54.690774 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:54.691114 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:54.694542 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "d7b72555c0554619b893618db30e32ec"
format_stamp: "Formatted at 2026-09-29 15:45:54 on dist-test-slave-7491"
server_key: "4501820131544aeb78355bad9da829c4"
server_key_iv: "6a65c8db93a3d4cebe5925dff0c23341"
server_key_version: "encryptionkey@0"
I20260929 15:45:54.695438 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7b72555c0554619b893618db30e32ec"
format_stamp: "Formatted at 2026-09-29 15:45:54 on dist-test-slave-7491"
server_key: "4501820131544aeb78355bad9da829c4"
server_key_iv: "6a65c8db93a3d4cebe5925dff0c23341"
server_key_version: "encryptionkey@0"
I20260929 15:45:54.702076 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 15:45:54.707302 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:54.709547 10743 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 15:45:54.709851 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "d7b72555c0554619b893618db30e32ec"
format_stamp: "Formatted at 2026-09-29 15:45:54 on dist-test-slave-7491"
server_key: "4501820131544aeb78355bad9da829c4"
server_key_iv: "6a65c8db93a3d4cebe5925dff0c23341"
server_key_version: "encryptionkey@0"
I20260929 15:45:54.710140 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:54.745416 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:54.751807 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:54.752208 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:54.776113 10743 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33325
I20260929 15:45:54.776254 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33325 every 8 connection(s)
I20260929 15:45:54.777696 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:45:54.781481 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10743
I20260929 15:45:54.781720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:45:54.782284 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:54.782541 388 external_mini_cluster.cc:1428] Setting key 6f2ba82b1b7e60c1521f7187b78203ee
I20260929 15:45:54.796773 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec: Bootstrap starting.
Sep 29 15:45:54 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:54.801923 10810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:54.803442 10810 log.cc:824] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:54.807364 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec: No bootstrap required, opened a new log
I20260929 15:45:54.809744 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:54.785358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.97.62:33325)
0929 15:45:54.786302 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:54.786316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:54.786364 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:45:54.788309 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:54.788337 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:54.788841 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:54.789333 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:54.790865 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:54.792226 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:54.793574 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:54.794421 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:54.794656 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:54.799486 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:54.799498 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:54.799525 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:54.799595 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:54.803761 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:54.804912 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:54.804919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:54.804928 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:54.805133 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:54.805872 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:54.805877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:54.805880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:54.806714 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:54.807267 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:54.807887 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 15:45:54.808524 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":440,"threads_started":1}
I20260929 15:45:54.817728 10810 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b72555c0554619b893618db30e32ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33325 } }
I20260929 15:45:54.818240 10810 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:54.818359 10810 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b72555c0554619b893618db30e32ec, State: Initialized, Role: FOLLOWER
I20260929 15:45:54.819051 10810 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b72555c0554619b893618db30e32ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33325 } }
I20260929 15:45:54.819628 10810 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:45:54.819868 10810 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:45:54.820173 10810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:54.823365 10810 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b72555c0554619b893618db30e32ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33325 } }
I20260929 15:45:54.824157 10810 leader_election.cc:302] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7b72555c0554619b893618db30e32ec; no voters:
I20260929 15:45:54.825131 10810 leader_election.cc:288] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:45:54.825300 10815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:54.826692 10815 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [term 1 LEADER]: Becoming Leader. State: Replica: d7b72555c0554619b893618db30e32ec, State: Running, Role: LEADER
I20260929 15:45:54.827452 10815 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b72555c0554619b893618db30e32ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33325 } }
I20260929 15:45:54.828114 10810 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:45:54.830902 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b72555c0554619b893618db30e32ec. Latest consensus state: current_term: 1 leader_uuid: "d7b72555c0554619b893618db30e32ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b72555c0554619b893618db30e32ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33325 } } }
I20260929 15:45:54.831547 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7b72555c0554619b893618db30e32ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b72555c0554619b893618db30e32ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33325 } } }
I20260929 15:45:54.832264 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:54.832334 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec [sys.catalog]: This master's current role is: LEADER
I20260929 15:45:54.834465 10820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:45:54.840644 10820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:45:54.851085 10820 catalog_manager.cc:1347] Generated new cluster ID: 8d0b7371732443dc80ed6c2355ba85af
I20260929 15:45:54.851274 10820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:45:54.866542 10820 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:45:54.868088 10820 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:45:54.881584 10820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec: Generated new TSK 0
I20260929 15:45:54.882658 10820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:45:54.956820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33325
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:55.356272 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:55.356616 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:55.356717 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:55.366225 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:55.366453 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:55.366595 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:45:55.379002 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:55.380741 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:55.382149 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:55.397507 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:55.397593 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:55.397593 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:55.398622 10836 server_base.cc:1034] running on GCE node
I20260929 15:45:55.399259 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:55.400379 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:55.401628 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790696755401586 us; error 45 us; skew 500 ppm
I20260929 15:45:55.402022 10836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:55 dist-test-slave-7491 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:55.405962 10836 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:45:55.407948 10836 webserver.cc:460] Webserver started at http://127.0.97.1:43837/ using document root <none> and password file <none>
I20260929 15:45:55.408912 10836 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:55.409098 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:55.409474 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:55.413093 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "3597021df2864f848316df25b4bdceee"
format_stamp: "Formatted at 2026-09-29 15:45:55 on dist-test-slave-7491"
server_key: "6cc9037f08187e243090e4e0eac75088"
server_key_iv: "cbbd3757d19e0bd622e788095f2ac677"
server_key_version: "encryptionkey@0"
I20260929 15:45:55.414091 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3597021df2864f848316df25b4bdceee"
format_stamp: "Formatted at 2026-09-29 15:45:55 on dist-test-slave-7491"
server_key: "6cc9037f08187e243090e4e0eac75088"
server_key_iv: "cbbd3757d19e0bd622e788095f2ac677"
server_key_version: "encryptionkey@0"
I20260929 15:45:55.421370 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 15:45:55.426702 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:55.429003 10836 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:45:55.429270 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "3597021df2864f848316df25b4bdceee"
format_stamp: "Formatted at 2026-09-29 15:45:55 on dist-test-slave-7491"
server_key: "6cc9037f08187e243090e4e0eac75088"
server_key_iv: "cbbd3757d19e0bd622e788095f2ac677"
server_key_version: "encryptionkey@0"
I20260929 15:45:55.429558 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:55.456583 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:55.463907 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:55.464308 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:55.465482 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:55.467238 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:55.467335 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:55.467475 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:55.467547 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:55.510561 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36717
I20260929 15:45:55.510624 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36717 every 8 connection(s)
I20260929 15:45:55.512094 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:45:55.516698 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10836
I20260929 15:45:55.516944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:45:55.517694 388 external_mini_cluster.cc:1428] Setting key 46e329552232540e1abacecac0ed7aa2
Sep 29 15:45:55 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:55.539158 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:55.515331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34183 (local address 127.0.97.62:33325)
0929 15:45:55.516090 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:55.516103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:55.517243 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 15:45:55.519671 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:55.519676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:55.519768 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:55.519932 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:55.521836 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:55.523063 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:55.525644 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:55.526400 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:55.526608 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:55.531501 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:55.531507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:55.531510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:55.531551 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:55.534434 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:55.535607 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:55.535613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:55.535616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:55.535792 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:55.536723 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:55.536729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:55.536732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:55.537170 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:55.537370 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:55.538482 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0929 15:45:55.538765 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":390,"threads_started":1}
I20260929 15:45:55.540331 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:55.515982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33325 (local address 127.0.97.1:34183)
0929 15:45:55.517031 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:55.517082 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:55.519259 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:55.520327 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:55.520337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:55.520937 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:55.521598 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:55.521619 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:55.523424 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:55.523431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:55.525373 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:55.525399 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:55.526781 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:55.526790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:55.527080 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:55.528045 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:55.528075 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:55.531142 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:55.534622 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:55.534633 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:55.534654 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:55.535351 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:55.535958 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:55.535964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:55.535967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:55.536492 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:55.537341 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:55.537353 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:55.537928 (+ 575us) client_negotiation.cc:769] Sending connection context
0929 15:45:55.538309 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 15:45:55.538929 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":482,"threads_started":1}
I20260929 15:45:55.542865 10964 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33325
I20260929 15:45:55.543321 10964 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:55.544414 10964 heartbeater.cc:511] Master 127.0.97.62:33325 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:45:55.547307 10775 ts_manager.cc:194] Registered new tserver with Master: 3597021df2864f848316df25b4bdceee (127.0.97.1:36717)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:45:55.549033 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:45:55.580560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33325
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:45:55.989979 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:55.990309 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:55.990409 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:55.998309 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:55.998461 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:55.998600 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:45:56.009667 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:56.011076 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:56.012544 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:56.030383 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:56.030457 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:56.030490 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:56.031512 10970 server_base.cc:1034] running on GCE node
I20260929 15:45:56.032327 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:56.033473 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:56.034732 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790696756034684 us; error 77 us; skew 500 ppm
I20260929 15:45:56.035105 10970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:56 dist-test-slave-7491 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:56.039659 10970 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:45:56.041589 10970 webserver.cc:460] Webserver started at http://127.0.97.2:33685/ using document root <none> and password file <none>
I20260929 15:45:56.042779 10970 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:56.042994 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:56.043653 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:56.047176 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "89b14046e2054e349c115dc45a199886"
format_stamp: "Formatted at 2026-09-29 15:45:56 on dist-test-slave-7491"
server_key: "05d845e9562c3f86ab9fbabb404ba1e2"
server_key_iv: "a6eb938089523043853f7ddbd6bb04c5"
server_key_version: "encryptionkey@0"
I20260929 15:45:56.048127 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "89b14046e2054e349c115dc45a199886"
format_stamp: "Formatted at 2026-09-29 15:45:56 on dist-test-slave-7491"
server_key: "05d845e9562c3f86ab9fbabb404ba1e2"
server_key_iv: "a6eb938089523043853f7ddbd6bb04c5"
server_key_version: "encryptionkey@0"
I20260929 15:45:56.055454 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:45:56.061097 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:56.063520 10970 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260929 15:45:56.063823 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "89b14046e2054e349c115dc45a199886"
format_stamp: "Formatted at 2026-09-29 15:45:56 on dist-test-slave-7491"
server_key: "05d845e9562c3f86ab9fbabb404ba1e2"
server_key_iv: "a6eb938089523043853f7ddbd6bb04c5"
server_key_version: "encryptionkey@0"
I20260929 15:45:56.064301 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:56.081115 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:56.087337 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:56.087714 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:56.089005 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:56.090747 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:56.090879 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:56.091050 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:56.091130 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:56.131939 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33069
I20260929 15:45:56.132054 11097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33069 every 8 connection(s)
I20260929 15:45:56.133383 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:45:56.142539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 10970
I20260929 15:45:56.142711 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:45:56.143205 388 external_mini_cluster.cc:1428] Setting key 2ff26fc37c0615ac81b590916a618bc8
Sep 29 15:45:56 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:56.161233 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.136501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44773 (local address 127.0.97.62:33325)
0929 15:45:56.137131 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.137138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.138740 (+ 1602us) server_negotiation.cc:407] Connection header received
0929 15:45:56.141115 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.141121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.141250 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.141387 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:56.143853 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.144882 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.146807 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.147281 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.147431 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.152159 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:56.152233 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:56.152239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:56.152294 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:56.155966 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:56.157343 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:56.157351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:56.157355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:56.157701 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:56.158615 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:56.158623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:56.158631 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:56.158973 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:56.159160 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.160322 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.160771 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":375,"threads_started":1}
I20260929 15:45:56.162755 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.137334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33325 (local address 127.0.97.2:44773)
0929 15:45:56.138488 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.138539 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.140704 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.141698 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.141708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.142749 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:56.143627 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.143652 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.145067 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.145073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.146635 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.146649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.147560 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.147568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.147848 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.148919 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:56.148955 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:56.151781 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:56.156236 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:56.156250 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:56.156277 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:56.157154 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:56.157955 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:56.157962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:56.157966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:56.158313 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:56.159139 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:56.159150 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:56.159645 (+ 495us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.159977 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.161085 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":399,"threads_started":1}
I20260929 15:45:56.165093 11098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33325
I20260929 15:45:56.165522 11098 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:56.166685 11098 heartbeater.cc:511] Master 127.0.97.62:33325 requested a full tablet report, sending...
I20260929 15:45:56.168872 10775 ts_manager.cc:194] Registered new tserver with Master: 89b14046e2054e349c115dc45a199886 (127.0.97.2:33069)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:45:56.170019 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:45:56.205042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33325
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:45:56.553861 10964 heartbeater.cc:503] Master 127.0.97.62:33325 was elected leader, sending a full tablet report...
W20260929 15:45:56.611747 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:45:56.612111 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:45:56.612318 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:45:56.621021 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:45:56.621297 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:45:56.621707 11104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:45:56.634037 11104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:45:56.635620 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:45:56.637265 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:45:56.652249 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:56.652122 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:45:56.652113 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:45:56.653203 11104 server_base.cc:1034] running on GCE node
I20260929 15:45:56.653893 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:45:56.655061 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:45:56.656599 11104 hybrid_clock.cc:648] HybridClock initialized: now 1790696756656494 us; error 121 us; skew 500 ppm
I20260929 15:45:56.657078 11104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:45:56 dist-test-slave-7491 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:45:56.661260 11104 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:45:56.663204 11104 webserver.cc:460] Webserver started at http://127.0.97.3:43195/ using document root <none> and password file <none>
I20260929 15:45:56.664054 11104 fs_manager.cc:359] Metadata directory not provided
I20260929 15:45:56.664256 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:45:56.664703 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:45:56.668978 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "10908b39af344fdfa2b2bc8f963d91ac"
format_stamp: "Formatted at 2026-09-29 15:45:56 on dist-test-slave-7491"
server_key: "15b4e018ec40b050b7259de6d8a57de1"
server_key_iv: "20fab783414bd7fdab14eaf07db4eebb"
server_key_version: "encryptionkey@0"
I20260929 15:45:56.670519 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "10908b39af344fdfa2b2bc8f963d91ac"
format_stamp: "Formatted at 2026-09-29 15:45:56 on dist-test-slave-7491"
server_key: "15b4e018ec40b050b7259de6d8a57de1"
server_key_iv: "20fab783414bd7fdab14eaf07db4eebb"
server_key_version: "encryptionkey@0"
I20260929 15:45:56.678380 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:45:56.683068 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:56.685503 11104 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:45:56.685792 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "10908b39af344fdfa2b2bc8f963d91ac"
format_stamp: "Formatted at 2026-09-29 15:45:56 on dist-test-slave-7491"
server_key: "15b4e018ec40b050b7259de6d8a57de1"
server_key_iv: "20fab783414bd7fdab14eaf07db4eebb"
server_key_version: "encryptionkey@0"
I20260929 15:45:56.686100 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:45:56.711093 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:45:56.716266 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:45:56.716617 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:45:56.717726 11104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:45:56.719368 11104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:45:56.719487 11104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:56.719648 11104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:45:56.719726 11104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:45:56.759723 11104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42787
I20260929 15:45:56.759765 11231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42787 every 8 connection(s)
I20260929 15:45:56.761247 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:45:56.765038 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 11104
I20260929 15:45:56.765223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:45:56.765667 388 external_mini_cluster.cc:1428] Setting key 3f9eca32c66a9a7a9d0fb7ccf28f57cb
Sep 29 15:45:56 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:45:56.787921 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.764804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56171 (local address 127.0.97.62:33325)
0929 15:45:56.765330 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.765338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.766600 (+ 1262us) server_negotiation.cc:407] Connection header received
0929 15:45:56.769232 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.769242 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.769334 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.769440 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:56.771459 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.772327 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.774104 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.774645 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.774774 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.779187 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:56.779194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:56.779200 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:56.779242 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:56.783809 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:56.784901 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:56.784907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:56.784911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:56.785060 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:56.785580 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:56.785585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:56.785589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:56.785943 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:56.786091 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.787361 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.787603 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260929 15:45:56.789543 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.765485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33325 (local address 127.0.97.3:56171)
0929 15:45:56.766432 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.766481 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.768779 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.769810 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.769822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.770509 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:45:56.771214 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.771263 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.772526 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.772531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.773903 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.773918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.774949 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.774955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.775219 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.776141 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:45:56.776226 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:45:56.778919 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:45:56.784067 (+ 5148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:56.784077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:56.784093 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:56.784772 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:56.785244 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:45:56.785247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:45:56.785250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:45:56.785452 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:45:56.786087 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:45:56.786094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:45:56.786795 (+ 701us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.787148 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.787714 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":418,"threads_started":1}
I20260929 15:45:56.791803 11232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33325
I20260929 15:45:56.792348 11232 heartbeater.cc:464] Registering TS with master...
I20260929 15:45:56.793329 11232 heartbeater.cc:511] Master 127.0.97.62:33325 requested a full tablet report, sending...
I20260929 15:45:56.795183 10775 ts_manager.cc:194] Registered new tserver with Master: 10908b39af344fdfa2b2bc8f963d91ac (127.0.97.3:42787)
I20260929 15:45:56.796277 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56171
I20260929 15:45:56.808332 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:45:56.828398 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.813430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.0.97.62:33325)
0929 15:45:56.813761 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.813768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.814077 (+ 309us) server_negotiation.cc:407] Connection header received
0929 15:45:56.814307 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.814314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.814387 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.814499 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:56.815939 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.817120 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.818475 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.818946 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.819097 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.822128 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:56.822135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:56.822138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:56.822195 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:56.825366 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:56.826200 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:56.826206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:56.826211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:56.826345 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:56.826754 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:56.826761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:56.826764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:56.827103 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:56.827386 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.827738 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.827984 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 15:45:56.833130 10775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:45:56.841889 10775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:45:56.867942 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.855902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.97.1:36717)
0929 15:45:56.859104 (+ 3202us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.859114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.859146 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:45:56.859258 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.859266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.859572 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.860578 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:56.860846 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.862580 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.865125 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.866679 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.866841 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.867102 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.867407 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.867574 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3037,"thread_start_us":2893,"threads_started":1}
I20260929 15:45:56.867998 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.855370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36717 (local address 127.0.0.1:38786)
0929 15:45:56.857154 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.857216 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.857457 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.859894 (+ 2437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.859902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.859935 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:56.860416 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.860429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.862708 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.862713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.864944 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.864958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.866892 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.866898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.867110 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.867170 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.867345 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.867565 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":162,"spinlock_wait_cycles":18176,"thread_start_us":1214,"threads_started":1}
I20260929 15:45:56.869292 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.859267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.0.97.2:33069)
0929 15:45:56.860112 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.860120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.860143 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:45:56.860310 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.860318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.860532 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.860741 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:56.861638 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.863123 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.866146 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.867693 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.867830 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.868018 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.868757 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.868965 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":557,"threads_started":1}
I20260929 15:45:56.869349 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.858736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33069 (local address 127.0.0.1:47550)
0929 15:45:56.859840 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.859862 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.859986 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.860997 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.861001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.861025 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:56.861447 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.861459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.863269 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.863276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.865938 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.865954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.868278 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.868285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.868482 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.868516 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.868612 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.868771 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":109,"thread_start_us":819,"threads_started":1}
I20260929 15:45:56.869930 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.855579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42787 (local address 127.0.0.1:41388)
0929 15:45:56.858369 (+ 2790us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.858389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.860102 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.860755 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.860759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.860782 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:56.861237 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.861249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.863139 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.863146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.865938 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.865954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.868371 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.868378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.868640 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.868675 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.868765 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.868930 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2662,"mutex_wait_us":31,"thread_start_us":2507,"threads_started":1}
I20260929 15:45:56.871070 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.856350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.97.3:42787)
0929 15:45:56.857857 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.857871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.858482 (+ 611us) server_negotiation.cc:407] Connection header received
0929 15:45:56.860309 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.860318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.860535 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.860741 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:56.861489 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.863001 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.866142 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.867693 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.867830 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.868627 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.869404 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.870688 (+ 1284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1303,"thread_start_us":1132,"threads_started":1}
I20260929 15:45:56.872992 10899 tablet_service.cc:1467] Processing CreateTablet for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 (DEFAULT_TABLE table=test-table [id=ddb9492189e24212a778d81710788623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:56.872991 11033 tablet_service.cc:1467] Processing CreateTablet for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 (DEFAULT_TABLE table=test-table [id=ddb9492189e24212a778d81710788623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:56.874392 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:56.874428 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:56.874945 11167 tablet_service.cc:1467] Processing CreateTablet for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 (DEFAULT_TABLE table=test-table [id=ddb9492189e24212a778d81710788623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:45:56.876411 11167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:45:56.886830 11251 tablet_bootstrap.cc:492] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Bootstrap starting.
I20260929 15:45:56.890726 11252 tablet_bootstrap.cc:492] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Bootstrap starting.
I20260929 15:45:56.892201 11251 tablet_bootstrap.cc:654] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:56.893357 11253 tablet_bootstrap.cc:492] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Bootstrap starting.
I20260929 15:45:56.893692 11251 log.cc:824] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:56.894812 11252 tablet_bootstrap.cc:654] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:56.896126 11252 log.cc:824] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:56.897044 11253 tablet_bootstrap.cc:654] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Neither blocks nor log segments found. Creating new log.
I20260929 15:45:56.897394 11251 tablet_bootstrap.cc:492] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: No bootstrap required, opened a new log
I20260929 15:45:56.897728 11251 ts_tablet_manager.cc:1397] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260929 15:45:56.898316 11253 log.cc:824] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Log is configured to *not* fsync() on all Append() calls
I20260929 15:45:56.900234 11252 tablet_bootstrap.cc:492] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: No bootstrap required, opened a new log
I20260929 15:45:56.900550 11252 ts_tablet_manager.cc:1397] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20260929 15:45:56.901762 11253 tablet_bootstrap.cc:492] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: No bootstrap required, opened a new log
I20260929 15:45:56.902067 11253 ts_tablet_manager.cc:1397] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 15:45:56.909909 11251 raft_consensus.cc:348] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.910650 11251 raft_consensus.cc:374] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:56.910768 11251 raft_consensus.cc:729] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3597021df2864f848316df25b4bdceee, State: Initialized, Role: FOLLOWER
I20260929 15:45:56.911517 11251 consensus_queue.cc:260] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.911408 11253 raft_consensus.cc:348] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.912130 11253 raft_consensus.cc:374] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:56.912292 11253 raft_consensus.cc:729] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10908b39af344fdfa2b2bc8f963d91ac, State: Initialized, Role: FOLLOWER
I20260929 15:45:56.912138 11252 raft_consensus.cc:348] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.912863 11252 raft_consensus.cc:374] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:45:56.912990 11252 raft_consensus.cc:729] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b14046e2054e349c115dc45a199886, State: Initialized, Role: FOLLOWER
I20260929 15:45:56.913290 11253 consensus_queue.cc:260] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.914058 11252 consensus_queue.cc:260] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.915198 11232 heartbeater.cc:503] Master 127.0.97.62:33325 was elected leader, sending a full tablet report...
I20260929 15:45:56.915863 11253 ts_tablet_manager.cc:1428] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260929 15:45:56.917121 11098 heartbeater.cc:503] Master 127.0.97.62:33325 was elected leader, sending a full tablet report...
I20260929 15:45:56.918742 11252 ts_tablet_manager.cc:1428] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260929 15:45:56.921123 11251 ts_tablet_manager.cc:1428] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Time spent starting tablet: real 0.023s user 0.008s sys 0.016s
I20260929 15:45:56.950189 11259 raft_consensus.cc:482] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:45:56.950549 11259 raft_consensus.cc:504] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.952491 11259 leader_election.cc:288] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3597021df2864f848316df25b4bdceee (127.0.97.1:36717), 10908b39af344fdfa2b2bc8f963d91ac (127.0.97.3:42787)
I20260929 15:45:56.963598 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.952923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59167 (local address 127.0.97.3:42787)
0929 15:45:56.953379 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.953389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.953796 (+ 407us) server_negotiation.cc:407] Connection header received
0929 15:45:56.954024 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.954032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.954117 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.954221 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:56.955209 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.956561 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.960506 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.962003 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.962178 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.962296 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.963156 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.963307 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260929 15:45:56.963598 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.953102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40377 (local address 127.0.97.1:36717)
0929 15:45:56.953570 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.953575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.953952 (+ 377us) server_negotiation.cc:407] Connection header received
0929 15:45:56.954096 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.954102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.954181 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.954263 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:45:56.955209 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.956565 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.960503 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.962003 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.962178 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.962296 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:56.963156 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 15:45:56.963307 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260929 15:45:56.964617 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.952916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36717 (local address 127.0.97.2:40377)
0929 15:45:56.953836 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.953851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.953960 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.954320 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.954329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.954357 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:56.954886 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.954904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.956818 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.956825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.960201 (+ 3376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.960228 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.962560 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.962568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.962824 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.962978 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.963875 (+ 897us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.964029 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":326,"thread_start_us":583,"threads_started":1}
I20260929 15:45:56.964640 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.952774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42787 (local address 127.0.97.2:59167)
0929 15:45:56.953626 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 15:45:56.953654 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:45:56.953842 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:45:56.954320 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:45:56.954326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:45:56.954425 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:45:56.954925 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.954939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.956819 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.956825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.960201 (+ 3376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:45:56.960228 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.962564 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:45:56.962569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:45:56.962824 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:56.962977 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 15:45:56.963828 (+ 851us) client_negotiation.cc:240] Negotiation successful
0929 15:45:56.964029 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":329,"spinlock_wait_cycles":27008,"thread_start_us":426,"threads_started":1}
I20260929 15:45:56.965849 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b6af8e9b9b4c7eb1e60acc3bf7d780" candidate_uuid: "89b14046e2054e349c115dc45a199886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3597021df2864f848316df25b4bdceee" is_pre_election: true
I20260929 15:45:56.965906 11187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b6af8e9b9b4c7eb1e60acc3bf7d780" candidate_uuid: "89b14046e2054e349c115dc45a199886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10908b39af344fdfa2b2bc8f963d91ac" is_pre_election: true
I20260929 15:45:56.966432 10919 raft_consensus.cc:2454] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b14046e2054e349c115dc45a199886 in term 0.
I20260929 15:45:56.966454 11187 raft_consensus.cc:2454] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b14046e2054e349c115dc45a199886 in term 0.
I20260929 15:45:56.967295 10987 leader_election.cc:302] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10908b39af344fdfa2b2bc8f963d91ac, 3597021df2864f848316df25b4bdceee, 89b14046e2054e349c115dc45a199886; no voters:
I20260929 15:45:56.967974 11259 raft_consensus.cc:2781] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:45:56.968262 11259 raft_consensus.cc:482] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:45:56.968384 11259 raft_consensus.cc:3037] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:56.971215 11259 raft_consensus.cc:504] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.971994 11259 leader_election.cc:288] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [CANDIDATE]: Term 1 election: Requested vote from peers 3597021df2864f848316df25b4bdceee (127.0.97.1:36717), 10908b39af344fdfa2b2bc8f963d91ac (127.0.97.3:42787)
I20260929 15:45:56.972803 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b6af8e9b9b4c7eb1e60acc3bf7d780" candidate_uuid: "89b14046e2054e349c115dc45a199886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3597021df2864f848316df25b4bdceee"
I20260929 15:45:56.972931 11187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b6af8e9b9b4c7eb1e60acc3bf7d780" candidate_uuid: "89b14046e2054e349c115dc45a199886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10908b39af344fdfa2b2bc8f963d91ac"
I20260929 15:45:56.973045 10919 raft_consensus.cc:3037] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:56.973387 11187 raft_consensus.cc:3037] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:45:56.975363 10919 raft_consensus.cc:2454] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89b14046e2054e349c115dc45a199886 in term 1.
I20260929 15:45:56.975921 10988 leader_election.cc:302] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3597021df2864f848316df25b4bdceee, 89b14046e2054e349c115dc45a199886; no voters:
I20260929 15:45:56.976022 11187 raft_consensus.cc:2454] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89b14046e2054e349c115dc45a199886 in term 1.
I20260929 15:45:56.976315 11259 raft_consensus.cc:2781] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:45:56.976823 11259 raft_consensus.cc:686] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 1 LEADER]: Becoming Leader. State: Replica: 89b14046e2054e349c115dc45a199886, State: Running, Role: LEADER
I20260929 15:45:56.977341 11259 consensus_queue.cc:237] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } }
I20260929 15:45:56.983397 10774 catalog_manager.cc:5495] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89b14046e2054e349c115dc45a199886 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "89b14046e2054e349c115dc45a199886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89b14046e2054e349c115dc45a199886" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } health_report { overall_health: HEALTHY } } }
I20260929 15:45:57.003561 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:56.997061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.0.97.2:33069)
0929 15:45:56.997344 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:56.997349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:56.997373 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:45:56.997474 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:56.997478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:56.997563 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:56.997724 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:45:56.998290 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:56.999038 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.000687 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.001245 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.001364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:57.002082 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:45:57.002675 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:45:57.002829 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:57.003101 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 15:45:57.003196 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260929 15:45:57.013042 11233 tablet.cc:2369] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:57.014736 10919 raft_consensus.cc:1261] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 1 FOLLOWER]: Refusing update from remote peer 89b14046e2054e349c115dc45a199886: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:45:57.014778 11187 raft_consensus.cc:1261] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 1 FOLLOWER]: Refusing update from remote peer 89b14046e2054e349c115dc45a199886: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 15:45:57.014981 10965 tablet.cc:2369] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:45:57.016031 11262 consensus_queue.cc:1035] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3597021df2864f848316df25b4bdceee" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:57.016522 11259 consensus_queue.cc:1035] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10908b39af344fdfa2b2bc8f963d91ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:45:57.048894 10774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:57.049249 10774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:45:57.053514 10774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7b72555c0554619b893618db30e32ec: Sending DeleteTablet for 3 replicas of tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780
I20260929 15:45:57.054991 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38786
I20260929 15:45:57.055394 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47550
I20260929 15:45:57.055809 11167 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:45:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41388
I20260929 15:45:57.057284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10836
I20260929 15:45:57.060798 11276 tablet_replica.cc:330] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: stopping tablet replica
I20260929 15:45:57.061444 11276 raft_consensus.cc:2229] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:45:57.061753 11277 tablet_replica.cc:330] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: stopping tablet replica
I20260929 15:45:57.062227 11277 raft_consensus.cc:2229] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:57.062637 11276 raft_consensus.cc:2258] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:57.062844 11277 raft_consensus.cc:2258] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:57.063733 11275 tablet_replica.cc:330] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: stopping tablet replica
I20260929 15:45:57.064247 11275 raft_consensus.cc:2229] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:45:57.064812 11275 raft_consensus.cc:2258] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:45:57.065487 11276 ts_tablet_manager.cc:1905] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:57.067472 11275 ts_tablet_manager.cc:1905] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:57.068627 11277 ts_tablet_manager.cc:1905] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:45:57.074870 11276 ts_tablet_manager.cc:1918] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 29 15:45:57 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:45:57.075138 11276 log.cc:1196] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/11b6af8e9b9b4c7eb1e60acc3bf7d780
I20260929 15:45:57.075899 11276 ts_tablet_manager.cc:1939] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 89b14046e2054e349c115dc45a199886: Deleting consensus metadata
I20260929 15:45:57.078241 10761 catalog_manager.cc:4841] TS 89b14046e2054e349c115dc45a199886 (127.0.97.2:33069): tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 (table test-table [id=ddb9492189e24212a778d81710788623]) successfully deleted
I20260929 15:45:57.080516 11277 ts_tablet_manager.cc:1918] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:57.080716 11277 log.cc:1196] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/11b6af8e9b9b4c7eb1e60acc3bf7d780
I20260929 15:45:57.081347 11277 ts_tablet_manager.cc:1939] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 10908b39af344fdfa2b2bc8f963d91ac: Deleting consensus metadata
I20260929 15:45:57.083500 10760 catalog_manager.cc:4841] TS 10908b39af344fdfa2b2bc8f963d91ac (127.0.97.3:42787): tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 (table test-table [id=ddb9492189e24212a778d81710788623]) successfully deleted
I20260929 15:45:57.085117 11275 ts_tablet_manager.cc:1918] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:45:57.085172 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:57.058639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.97.1:36717)
0929 15:45:57.058847 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:57.058853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:57.061028 (+ 2175us) server_negotiation.cc:407] Connection header received
0929 15:45:57.061239 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:57.061245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:57.061339 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:57.061461 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:57.065388 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.066582 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.068697 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.069395 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.069503 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:57.076555 (+ 7052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:57.076578 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:57.076605 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:57.076677 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:57.081153 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:57.082249 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:57.082266 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:57.082270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:57.082466 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:57.082955 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:57.082965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:57.082968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:57.083474 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:57.083821 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:57.084443 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 15:45:57.084717 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 15:45:57.085359 11275 log.cc:1196] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/11b6af8e9b9b4c7eb1e60acc3bf7d780
I20260929 15:45:57.085971 11275 ts_tablet_manager.cc:1939] T 11b6af8e9b9b4c7eb1e60acc3bf7d780 P 3597021df2864f848316df25b4bdceee: Deleting consensus metadata
I20260929 15:45:57.086151 10959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:57.187552 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10836
I20260929 15:45:57.187686 10761 catalog_manager.cc:4841] TS 3597021df2864f848316df25b4bdceee (127.0.97.1:36717): tablet 11b6af8e9b9b4c7eb1e60acc3bf7d780 (table test-table [id=ddb9492189e24212a778d81710788623]) successfully deleted
I20260929 15:45:57.207540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10970
Sep 29 15:45:57 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:45:57.230819 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:57.208384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.0.97.2:33069)
0929 15:45:57.208614 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:57.208619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:57.208696 (+ 77us) server_negotiation.cc:407] Connection header received
0929 15:45:57.209041 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:57.209049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:57.209177 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:57.209332 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:57.211110 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.212004 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.214016 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.214642 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.214762 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:57.221775 (+ 7013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:57.221799 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:57.221828 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:57.221894 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:57.225939 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:57.227322 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:57.227336 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:57.227339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:57.227647 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:57.228345 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:57.228362 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:57.228366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:57.228900 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:57.229230 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:57.229857 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 15:45:57.230424 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 15:45:57.231835 11093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:57.353695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10970
I20260929 15:45:57.378708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 11104
Sep 29 15:45:57 dist-test-slave-7491 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:45:57.398365 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:45:57.379813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.97.3:42787)
0929 15:45:57.380112 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 15:45:57.380120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:45:57.380303 (+ 183us) server_negotiation.cc:407] Connection header received
0929 15:45:57.380532 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:45:57.380537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:45:57.380627 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:45:57.380747 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:45:57.382314 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.383085 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.384914 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:45:57.385363 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:45:57.385500 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:45:57.391286 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:45:57.391304 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:45:57.391322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:45:57.391367 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:45:57.395335 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:57.396268 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:57.396276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:57.396279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:57.396426 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:45:57.396794 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:45:57.396799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:45:57.396801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:45:57.397194 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:45:57.397428 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 15:45:57.397783 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 15:45:57.397966 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 15:45:57.399173 11227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:57.495827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 11104
I20260929 15:45:57.514025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 10743
I20260929 15:45:57.515528 10805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:45:57.640035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 10743
2026-09-29T15:45:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4838 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:45:57 dist-test-slave-7491 krb5kdc[11285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:45:57 dist-test-slave-7491 krb5kdc[11285](info): set up 2 sockets
Sep 29 15:45:57 dist-test-slave-7491 krb5kdc[11285](info): commencing operation
krb5kdc: starting...
W20260929 15:45:58.830163 388 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:45:59 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:45:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 15:45:59.262017 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQOquZz/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/postgres -l logfile start
2026-09-29 15:46:02.325 UTC [11315] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:46:02.325 UTC [11315] LOG: listening on IPv4 address "127.0.97.20", port 59715
2026-09-29 15:46:02.329 UTC [11315] LOG: listening on Unix socket "/tmp/.s.PGSQL.59715"
2026-09-29 15:46:02.355 UTC [11318] LOG: database system was shut down at 2026-09-29 15:46:00 UTC
2026-09-29 15:46:02.360 UTC [11315] LOG: database system is ready to accept connections
I20260929 15:46:03.376983 388 mini_postgres.cc:96] Postgres bound to 59715
2026-09-29 15:46:03.397 UTC [11326] FATAL: database "slave" does not exist
127.0.97.20:59715 - accepting connections
I20260929 15:46:03.398459 388 mini_ranger.cc:161] Starting Ranger...
I20260929 15:46:03.429313 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:46:04.059053 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:46:04.059243 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-admin
2026-09-29 15:46:04,408 [I] DB FLAVOR :POSTGRES
2026-09-29 15:46:04,409 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:46:04,409 [I] Checking connection..
2026-09-29 15:46:04,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:46:04,732 [I] Checking connection passed.
2026-09-29 15:46:04,733 [I] --------- Verifying version history table ---------
2026-09-29 15:46:04,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:46:05,051 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:46:05,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:46:05,355 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:46:05,356 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:46:05,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:46:05.660 UTC [11390] WARNING: there is no transaction in progress
2026-09-29 15:46:05,668 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:46:05,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:46:05,969 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:46:05,969 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:46:05,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:46:06,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:46:06,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7491.c.gcp-upstream.internal','N') ;"
2026-09-29 15:46:06,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:46:07,147 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:46:07,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:46:07,453 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:46:07,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:46:07,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:46:08,058 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:46:08,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:46:09.275 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.295 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.316 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.337 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.357 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.381 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.460 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.470 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.489 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.502 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.516 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.533 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.546 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.555 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.565 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:09.575 UTC [11498] WARNING: there is no transaction in progress
2026-09-29 15:46:10,483 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:46:10,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:46:10,784 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:46:10,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:46:11,088 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:46:11,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:46:11,378 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:46:11,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:46:11,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:46:11,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7491.c.gcp-upstream.internal';"
2026-09-29 15:46:12,284 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:46:12,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:46:12,574 [I] DB_PATCHES have already been applied
I20260929 15:46:12.580459 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQOquZz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:46:12.580615 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 15:46:12.600016 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:33121
Sep 29, 2026 3:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:46:13.819792 388 mini_ranger.cc:160] Time spent starting Ranger: real 10.421s user 0.000s sys 0.050s
Sep 29, 2026 3:46:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:46:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 3:46:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:46:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:46:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 15:46:41.066864 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 15:46:41 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696801, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 15:46:41.283265 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 15:46:41.310211 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 15:46:42.324204 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 15:46:42.324364 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms
I20260929 15:46:42.324524 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:59715
2026-09-29 15:46:42,813 [I] DB FLAVOR :POSTGRES
2026-09-29 15:46:42,813 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:46:42,814 [I] Checking connection
2026-09-29 15:46:42,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 15:46:43,099 [I] connection success
2026-09-29 15:46:43,100 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 15:46:43,100 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 15:46:43,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 15:46:43,377 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 15:46:43,377 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:46:43,377 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 15:46:43,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:59715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 15:46:43,671 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 15:46:44.221172 388 mini_ranger_kms.cc:310] Created kms service
I20260929 15:46:44.455324 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 15:46:44.517302 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 15:46:44 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696801, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:46:44.965622 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 15:46:44.965976 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 15:46:44.966042 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 15:46:44.985271 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11679
I20260929 15:46:44.985491 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:36401
Sep 29, 2026 3:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:46:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:46:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:46:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:46:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 15:46:47 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790696807, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:46:47.374850 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.092s user 0.148s sys 0.353s
I20260929 15:46:47.375094 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 15:46:47.375308 388 mini_ranger_kms.cc:401] 127.0.97.20:36401/kms/v1/keys
Sep 29 15:46:47 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696807, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:46:47 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696807, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:46:50 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:46:50.729529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40303
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40303
--ranger_config_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36401/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:46:51.136431 11729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:46:51.136780 11729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:46:51.136870 11729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:46:51.144699 11729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:46:51.144838 11729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:46:51.144888 11729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:46:51.144938 11729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:46:51.144980 11729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:46:51.156623 11729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:36401/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40303
--ranger_config_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:46:51.157950 11729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:46:51.159298 11729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:46:51.173293 11737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:51.173354 11735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:46:51.175518 11729 server_base.cc:1034] running on GCE node
W20260929 15:46:51.175962 11734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:46:51.177122 11729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:46:51.178956 11729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:46:51.180411 11729 hybrid_clock.cc:648] HybridClock initialized: now 1790696811180316 us; error 52 us; skew 500 ppm
I20260929 15:46:51.180997 11729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:46:51 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:46:51.185254 11729 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:46:51.187616 11729 webserver.cc:460] Webserver started at http://127.0.97.62:34557/ using document root <none> and password file <none>
I20260929 15:46:51.188714 11729 fs_manager.cc:359] Metadata directory not provided
I20260929 15:46:51.188889 11729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:46:51.189229 11729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:46:51 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:46:51.254863 11729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "b33a358921e440d5bf19ee55a6ad0cb4"
format_stamp: "Formatted at 2026-09-29 15:46:51 on dist-test-slave-7491"
server_key: "689706c75d06569924357021cbf66cf5"
server_key_iv: "92464aff2212ef4de53166ed634009dd"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:51.255987 11729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "b33a358921e440d5bf19ee55a6ad0cb4"
format_stamp: "Formatted at 2026-09-29 15:46:51 on dist-test-slave-7491"
server_key: "689706c75d06569924357021cbf66cf5"
server_key_iv: "92464aff2212ef4de53166ed634009dd"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:51.263440 11729 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260929 15:46:51.286288 11744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:51.288612 11729 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260929 15:46:51.288852 11729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "b33a358921e440d5bf19ee55a6ad0cb4"
format_stamp: "Formatted at 2026-09-29 15:46:51 on dist-test-slave-7491"
server_key: "689706c75d06569924357021cbf66cf5"
server_key_iv: "92464aff2212ef4de53166ed634009dd"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:51.289130 11729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:46:51.311465 11729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:46:51.317688 11729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:46:51.318109 11729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:46:51.343390 11729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40303
I20260929 15:46:51.343525 11795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40303 every 8 connection(s)
I20260929 15:46:51.345081 11729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:46:51.351022 11796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:46:51.355314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 11729
I20260929 15:46:51.355561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:46:51.356357 388 external_mini_cluster.cc:1428] Setting key 42bd2ced772c7cb30e1f5a0be1dc46df
I20260929 15:46:51.369374 11796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4: Bootstrap starting.
I20260929 15:46:51.374068 11796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4: Neither blocks nor log segments found. Creating new log.
I20260929 15:46:51.375445 11796 log.cc:824] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:46:51 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:46:51.378845 11796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4: No bootstrap required, opened a new log
I20260929 15:46:51.386066 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:51.359362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.97.62:40303)
0929 15:46:51.360321 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:51.360335 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:51.360383 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:46:51.362734 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:51.362763 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:51.363285 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:51.363695 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:51.366379 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:51.367892 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:51.370367 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:51.371140 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:51.371555 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:51.377135 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:51.377145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:51.377164 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:51.377211 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:51.381086 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:51.382075 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:51.382082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:51.382095 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:51.382275 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:51.382706 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:51.382711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:51.382713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:51.383484 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:51.383866 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:51.384399 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 15:46:51.384878 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":411,"threads_started":1}
I20260929 15:46:51.389940 11796 raft_consensus.cc:348] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260929 15:46:51.390414 11796 raft_consensus.cc:374] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:46:51.390529 11796 raft_consensus.cc:729] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33a358921e440d5bf19ee55a6ad0cb4, State: Initialized, Role: FOLLOWER
I20260929 15:46:51.391197 11796 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260929 15:46:51.391458 11796 raft_consensus.cc:388] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:46:51.391604 11796 raft_consensus.cc:482] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:46:51.391790 11796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:46:51.394479 11796 raft_consensus.cc:504] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260929 15:46:51.395133 11796 leader_election.cc:302] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33a358921e440d5bf19ee55a6ad0cb4; no voters:
I20260929 15:46:51.395838 11796 leader_election.cc:288] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:46:51.395973 11801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:46:51.397341 11801 raft_consensus.cc:686] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [term 1 LEADER]: Becoming Leader. State: Replica: b33a358921e440d5bf19ee55a6ad0cb4, State: Running, Role: LEADER
I20260929 15:46:51.398011 11801 consensus_queue.cc:237] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260929 15:46:51.398772 11796 sys_catalog.cc:564] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:46:51.401398 11803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b33a358921e440d5bf19ee55a6ad0cb4. Latest consensus state: current_term: 1 leader_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } } }
I20260929 15:46:51.401376 11802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33a358921e440d5bf19ee55a6ad0cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } } }
I20260929 15:46:51.402479 11803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:46:51.402479 11802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:46:51.408052 11809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:46:51.414655 11809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:46:51.415619 11796 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 15:46:51.436105 11809 catalog_manager.cc:1347] Generated new cluster ID: a2012ca28ac544e5bf7f0ca372e8191a
I20260929 15:46:51.436504 11809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:46:51.456898 11809 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:46:51.458571 11809 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:46:51.467952 11809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4: Generated new TSK 0
I20260929 15:46:51.468997 11809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:46:52.304876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36401/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:46:52 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790696812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 15:46:52.745766 11841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:46:52.746120 11841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:46:52.746290 11841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:46:52.754858 11841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:46:52.755049 11841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:46:52.755244 11841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:46:52.769339 11841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:36401/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:46:52.771149 11841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:46:52.773063 11841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:46:52.795861 11852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:52.802948 11851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:52.804097 11854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:46:52.805056 11841 server_base.cc:1034] running on GCE node
I20260929 15:46:52.805892 11841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:46:52.815987 11841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:46:52.817443 11841 hybrid_clock.cc:648] HybridClock initialized: now 1790696812817365 us; error 118 us; skew 500 ppm
I20260929 15:46:52.817978 11841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:46:52 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:46:52.823035 11841 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:46:52.825807 11841 webserver.cc:460] Webserver started at http://127.0.97.1:34373/ using document root <none> and password file <none>
I20260929 15:46:52.826822 11841 fs_manager.cc:359] Metadata directory not provided
I20260929 15:46:52.826979 11841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:46:52.827409 11841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:46:52 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:46:52.856544 11841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "86ac403501d94e69b6d5fbb9bb910919"
format_stamp: "Formatted at 2026-09-29 15:46:52 on dist-test-slave-7491"
server_key: "71dec15962f45b8783a9903ae6df1468"
server_key_iv: "405af14ec6ab52b5d01950c1a4880cb3"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:52.858501 11841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "86ac403501d94e69b6d5fbb9bb910919"
format_stamp: "Formatted at 2026-09-29 15:46:52 on dist-test-slave-7491"
server_key: "71dec15962f45b8783a9903ae6df1468"
server_key_iv: "405af14ec6ab52b5d01950c1a4880cb3"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:52.872664 11841 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.010s sys 0.001s
I20260929 15:46:52.904481 11861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:52.906752 11841 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.000s
I20260929 15:46:52.907063 11841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "86ac403501d94e69b6d5fbb9bb910919"
format_stamp: "Formatted at 2026-09-29 15:46:52 on dist-test-slave-7491"
server_key: "71dec15962f45b8783a9903ae6df1468"
server_key_iv: "405af14ec6ab52b5d01950c1a4880cb3"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:52.907415 11841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:46:52.927605 11841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:46:52.934216 11841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:46:52.934616 11841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:46:52.935866 11841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:46:52.937521 11841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:46:52.937639 11841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:52.937773 11841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:46:52.937862 11841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:53.047911 11841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40881
I20260929 15:46:53.049818 11973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40881 every 8 connection(s)
I20260929 15:46:53.049978 11841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:46:53.053182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 11841
I20260929 15:46:53.053390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:46:53.054119 388 external_mini_cluster.cc:1428] Setting key 5bf4eb7348de71ada983ba10ccf53e42
Sep 29 15:46:53 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:46:53.096055 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:53.054837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40303 (local address 127.0.97.1:50211)
0929 15:46:53.064349 (+ 9512us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:53.064432 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:53.067722 (+ 3290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:53.073780 (+ 6058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:53.073793 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:53.074596 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:46:53.075558 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:53.075592 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.077361 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.077369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:53.079814 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:53.079839 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.081407 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.081419 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:53.081809 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:53.082747 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:46:53.082785 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:46:53.085918 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:46:53.090499 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:46:53.090512 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:46:53.090535 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:46:53.091272 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:46:53.091859 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:46:53.091865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:46:53.091869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:46:53.092143 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:46:53.093030 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:46:53.093039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:46:53.093708 (+ 669us) client_negotiation.cc:769] Sending connection context
0929 15:46:53.094032 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 15:46:53.094647 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7628,"thread_start_us":7816,"threads_started":1}
I20260929 15:46:53.097988 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:53.053912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50211 (local address 127.0.97.62:40303)
0929 15:46:53.072867 (+ 18955us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:53.072874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:53.072898 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:46:53.072970 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:53.072975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:53.073059 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:53.073334 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:53.075824 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.077103 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.080272 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.081089 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.082374 (+ 1285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:53.086306 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:53.086314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:53.086319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:53.086367 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:53.090236 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:53.091479 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:53.091485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:53.091489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:53.091673 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:53.092320 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:53.092326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:53.092329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:53.092868 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:53.093021 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:53.097376 (+ 4355us) server_negotiation.cc:299] Negotiation successful
0929 15:46:53.097648 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18814,"thread_start_us":18665,"threads_started":1}
I20260929 15:46:53.099738 11974 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40303
I20260929 15:46:53.100157 11974 heartbeater.cc:464] Registering TS with master...
I20260929 15:46:53.101181 11974 heartbeater.cc:511] Master 127.0.97.62:40303 requested a full tablet report, sending...
I20260929 15:46:53.104140 11761 ts_manager.cc:194] Registered new tserver with Master: 86ac403501d94e69b6d5fbb9bb910919 (127.0.97.1:40881)
I20260929 15:46:53.106346 11761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:46:53.136901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36401/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:46:53 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:46:53 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 15:46:53.638121 11980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:46:53.638635 11980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:46:53.638846 11980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:46:53.649610 11980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:46:53.649926 11980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:46:53.650228 11980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:46:53.666674 11980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:36401/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:46:53.668848 11980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:46:53.671211 11980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:46:53.704768 11987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:53.705518 11988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:53.713203 11990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:46:53.716642 11980 server_base.cc:1034] running on GCE node
I20260929 15:46:53.717770 11980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:46:53.719487 11980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:46:53.720750 11980 hybrid_clock.cc:648] HybridClock initialized: now 1790696813720635 us; error 113 us; skew 500 ppm
I20260929 15:46:53.721160 11980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:46:53 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:46:53.731539 11980 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:46:53.734351 11980 webserver.cc:460] Webserver started at http://127.0.97.2:32883/ using document root <none> and password file <none>
I20260929 15:46:53.735591 11980 fs_manager.cc:359] Metadata directory not provided
I20260929 15:46:53.735767 11980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:46:53.736212 11980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:46:53 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:46:53.760252 11980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c27632d6f3c4c438e24a286c8aaf7bf"
format_stamp: "Formatted at 2026-09-29 15:46:53 on dist-test-slave-7491"
server_key: "c80f7dc44b6698e6ab5b8d618365467e"
server_key_iv: "a739e9abd928f43459f02851562bb83b"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:53.761612 11980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c27632d6f3c4c438e24a286c8aaf7bf"
format_stamp: "Formatted at 2026-09-29 15:46:53 on dist-test-slave-7491"
server_key: "c80f7dc44b6698e6ab5b8d618365467e"
server_key_iv: "a739e9abd928f43459f02851562bb83b"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:53.776273 11980 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.008s
I20260929 15:46:53.812937 11997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:53.818635 11980 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.000s
I20260929 15:46:53.818895 11980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "6c27632d6f3c4c438e24a286c8aaf7bf"
format_stamp: "Formatted at 2026-09-29 15:46:53 on dist-test-slave-7491"
server_key: "c80f7dc44b6698e6ab5b8d618365467e"
server_key_iv: "a739e9abd928f43459f02851562bb83b"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:53.819198 11980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:46:53.862834 11980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:46:53.867653 11980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:46:53.868017 11980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:46:53.869220 11980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:46:53.870895 11980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:46:53.871007 11980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:53.871151 11980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:46:53.871230 11980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:53.925884 11980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35287
I20260929 15:46:53.925949 12116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35287 every 8 connection(s)
I20260929 15:46:53.928144 11980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:46:53.940892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 11980
I20260929 15:46:53.941150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:46:53.941900 388 external_mini_cluster.cc:1428] Setting key e22557ee614cb2cc8171a74ba94f6c54
Sep 29 15:46:53 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:46:54.007457 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:53.934095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49735 (local address 127.0.97.62:40303)
0929 15:46:53.973086 (+ 38991us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:53.973097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:53.979988 (+ 6891us) server_negotiation.cc:407] Connection header received
0929 15:46:53.980096 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:53.980105 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:53.980278 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:53.980409 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:53.987631 (+ 7222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.989090 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.992004 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.992796 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.992925 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:53.998329 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:53.998340 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:53.998348 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:53.998403 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:54.003012 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:54.004298 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:54.004306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:54.004310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:54.004482 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:54.005124 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:54.005132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:54.005137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:54.005538 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:54.005688 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.006822 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.007070 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38774,"thread_start_us":39017,"threads_started":1}
I20260929 15:46:54.008731 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:53.933150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40303 (local address 127.0.97.2:49735)
0929 15:46:53.975358 (+ 42208us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:53.975452 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:53.979272 (+ 3820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:53.986058 (+ 6786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:53.986074 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:53.986620 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:46:53.987377 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:53.987405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.989344 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.989353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:53.991775 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:53.991800 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:53.993347 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:53.993357 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:53.993723 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:53.994739 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:46:53.994777 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:46:53.997994 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:46:54.003324 (+ 5330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:46:54.003336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:46:54.003358 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:46:54.004083 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:46:54.004663 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:46:54.004672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:46:54.004677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:46:54.004965 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:46:54.005813 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:46:54.005823 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:46:54.006276 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.006696 (+ 420us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.007274 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41698,"thread_start_us":50520,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 15:46:54.010510 12117 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40303
I20260929 15:46:54.010865 12117 heartbeater.cc:464] Registering TS with master...
I20260929 15:46:54.011678 12117 heartbeater.cc:511] Master 127.0.97.62:40303 requested a full tablet report, sending...
I20260929 15:46:54.019553 11761 ts_manager.cc:194] Registered new tserver with Master: 6c27632d6f3c4c438e24a286c8aaf7bf (127.0.97.2:35287)
I20260929 15:46:54.020732 11761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260929 15:46:54.046516 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.101s user 0.001s sys 0.018s
I20260929 15:46:54.047518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36401/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:46:54.110898 11974 heartbeater.cc:503] Master 127.0.97.62:40303 was elected leader, sending a full tablet report...
W20260929 15:46:54.489079 12123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:46:54.489457 12123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:46:54.489750 12123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:46:54.498117 12123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:46:54.498297 12123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:46:54.498473 12123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:46:54.510216 12123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:36401/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:46:54.511684 12123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:46:54.513137 12123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:46:54.527482 12128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:54.527621 12129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:46:54.527755 12131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:46:54.529181 12123 server_base.cc:1034] running on GCE node
I20260929 15:46:54.529918 12123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:46:54.531004 12123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:46:54.532310 12123 hybrid_clock.cc:648] HybridClock initialized: now 1790696814532258 us; error 49 us; skew 500 ppm
I20260929 15:46:54.532687 12123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:46:54 dist-test-slave-7491 krb5kdc[11285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:46:54.536861 12123 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:46:54.538822 12123 webserver.cc:460] Webserver started at http://127.0.97.3:33095/ using document root <none> and password file <none>
I20260929 15:46:54.539580 12123 fs_manager.cc:359] Metadata directory not provided
I20260929 15:46:54.539743 12123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:46:54.540071 12123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:46:54 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:46:54.558507 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "a394d193d6ce4630a5e51d20f24e9de8"
format_stamp: "Formatted at 2026-09-29 15:46:54 on dist-test-slave-7491"
server_key: "a10b916a83b4782f3fda2aaffe9551a1"
server_key_iv: "baa560e8cb3a6ebe8406b6a3f2dff140"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:54.559743 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a394d193d6ce4630a5e51d20f24e9de8"
format_stamp: "Formatted at 2026-09-29 15:46:54 on dist-test-slave-7491"
server_key: "a10b916a83b4782f3fda2aaffe9551a1"
server_key_iv: "baa560e8cb3a6ebe8406b6a3f2dff140"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:54.567054 12123 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 15:46:54.587141 12138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:54.588918 12123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:46:54.589109 12123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "a394d193d6ce4630a5e51d20f24e9de8"
format_stamp: "Formatted at 2026-09-29 15:46:54 on dist-test-slave-7491"
server_key: "a10b916a83b4782f3fda2aaffe9551a1"
server_key_iv: "baa560e8cb3a6ebe8406b6a3f2dff140"
server_key_version: "kuduclusterkey@0"
I20260929 15:46:54.589352 12123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:46:54.610397 12123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:46:54.615142 12123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:46:54.615487 12123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:46:54.616585 12123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:46:54.618083 12123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:46:54.618161 12123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:54.618288 12123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:46:54.618355 12123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:46:54.658268 12123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40839
I20260929 15:46:54.658304 12250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40839 every 8 connection(s)
I20260929 15:46:54.659742 12123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:46:54.669102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 12123
I20260929 15:46:54.669351 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:46:54.669787 388 external_mini_cluster.cc:1428] Setting key 8b21bb40a99e520515f00085d4bf7b8b
Sep 29 15:46:54 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:46:54.682476 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.662390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44945 (local address 127.0.97.62:40303)
0929 15:46:54.662885 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.662893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.664028 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 15:46:54.666050 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.666055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.666149 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.666256 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:54.667925 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.668817 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.670880 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.671459 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.671609 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.676081 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:54.676086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:54.676090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:54.676128 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:54.679084 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:54.679983 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:54.679989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:54.679993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:54.680128 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:54.680677 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:54.680681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:54.680686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:54.680984 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:54.681141 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.681871 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.682111 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":216,"threads_started":1}
I20260929 15:46:54.683581 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.662992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40303 (local address 127.0.97.3:44945)
0929 15:46:54.663832 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:54.663882 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:54.665717 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:54.666531 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:54.666541 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:54.667051 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:46:54.667693 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.667713 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.669021 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.669026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.670696 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.670712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.671723 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.671731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.672002 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.672853 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:46:54.672878 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:46:54.675785 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:46:54.679311 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:46:54.679320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:46:54.679337 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:46:54.679839 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:46:54.680320 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:46:54.680331 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:46:54.680334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:46:54.680546 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:46:54.681129 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:46:54.681139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:46:54.681504 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.681785 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.682345 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":382,"threads_started":1}
I20260929 15:46:54.685321 12251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40303
I20260929 15:46:54.685678 12251 heartbeater.cc:464] Registering TS with master...
I20260929 15:46:54.686502 12251 heartbeater.cc:511] Master 127.0.97.62:40303 requested a full tablet report, sending...
I20260929 15:46:54.688037 11761 ts_manager.cc:194] Registered new tserver with Master: a394d193d6ce4630a5e51d20f24e9de8 (127.0.97.3:40839)
I20260929 15:46:54.688971 11761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44945
I20260929 15:46:54.699412 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:46:54.719636 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.704905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.97.62:40303)
0929 15:46:54.705112 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.705117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.705512 (+ 395us) server_negotiation.cc:407] Connection header received
0929 15:46:54.705733 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.705739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.705824 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.705943 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:54.707932 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.708763 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.710744 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.711209 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.711355 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.714228 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:54.714234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:54.714237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:54.714278 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:54.716906 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:54.717645 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:54.717651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:54.717654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:54.717785 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:54.718175 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:54.718181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:54.718184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:54.718473 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:54.718734 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.719066 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.719310 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:46:54.724768 11761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:46:54.732434 11761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:46:54.774156 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.747922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.0.97.3:40839)
0929 15:46:54.763245 (+ 15323us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.763258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.764497 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 15:46:54.764609 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.764618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.764876 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.765073 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:46:54.766724 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.768373 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.771287 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.772686 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.772826 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.773091 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.773581 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.773757 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15071,"thread_start_us":15353,"threads_started":1}
I20260929 15:46:54.774202 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.747631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40839 (local address 127.0.0.1:48888)
0929 15:46:54.763675 (+ 16044us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:54.763742 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:54.764020 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:54.765372 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:54.765379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:54.765423 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:46:54.766498 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.766511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.769000 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.769004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.771069 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.771090 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.773196 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.773201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.773344 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.773382 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.773536 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.773749 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15708,"mutex_wait_us":117,"spinlock_wait_cycles":23680,"thread_start_us":16583,"threads_started":1}
I20260929 15:46:54.775641 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.748464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.97.1:40881)
0929 15:46:54.759821 (+ 11357us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.759835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.764291 (+ 4456us) server_negotiation.cc:407] Connection header received
0929 15:46:54.764570 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.764578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.764857 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.765059 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:46:54.767203 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.768495 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.772271 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.773812 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.773948 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.774167 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.775131 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.775298 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11010,"thread_start_us":10854,"threads_started":1}
I20260929 15:46:54.775652 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.754767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35287 (local address 127.0.0.1:37368)
0929 15:46:54.763920 (+ 9153us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:54.763936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:54.764040 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:54.765373 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:54.765381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:54.765438 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:46:54.766954 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.766967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.768798 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.768805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.771954 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.771964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.774322 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.774328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.774918 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.774954 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.775051 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.775233 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9037,"mutex_wait_us":376,"spinlock_wait_cycles":40320,"thread_start_us":9191,"threads_started":1}
I20260929 15:46:54.776381 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.747865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:41966)
0929 15:46:54.763692 (+ 15827us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:54.763743 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:54.764352 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:54.765373 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:54.765381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:54.765431 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:46:54.766956 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.766967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.768622 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.768630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.771949 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.771964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.774794 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.774806 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.774950 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.774974 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.775051 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.775626 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15645,"mutex_wait_us":363,"spinlock_wait_cycles":24960,"thread_start_us":15524,"threads_started":1}
I20260929 15:46:54.776597 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.748511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.0.97.2:35287)
0929 15:46:54.764638 (+ 16127us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.764660 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.764684 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:46:54.764771 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.764776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.764956 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.765135 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:46:54.767160 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.768661 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.772291 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.773811 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.773948 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.774138 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.776047 (+ 1909us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.776265 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15957,"thread_start_us":16895,"threads_started":1}
I20260929 15:46:54.778640 12186 tablet_service.cc:1467] Processing CreateTablet for tablet 31b45f19cbd84d99913d6b0a9e5fbf64 (DEFAULT_TABLE table=test-table [id=ad3019572af64e7c8ead893d2d7130cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:46:54.779929 12186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b45f19cbd84d99913d6b0a9e5fbf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:46:54.780120 12052 tablet_service.cc:1467] Processing CreateTablet for tablet 31b45f19cbd84d99913d6b0a9e5fbf64 (DEFAULT_TABLE table=test-table [id=ad3019572af64e7c8ead893d2d7130cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:46:54.780470 11909 tablet_service.cc:1467] Processing CreateTablet for tablet 31b45f19cbd84d99913d6b0a9e5fbf64 (DEFAULT_TABLE table=test-table [id=ad3019572af64e7c8ead893d2d7130cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:46:54.781740 11909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b45f19cbd84d99913d6b0a9e5fbf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:46:54.781839 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b45f19cbd84d99913d6b0a9e5fbf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:46:54.792395 12270 tablet_bootstrap.cc:492] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Bootstrap starting.
I20260929 15:46:54.797199 12270 tablet_bootstrap.cc:654] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Neither blocks nor log segments found. Creating new log.
I20260929 15:46:54.798229 12271 tablet_bootstrap.cc:492] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Bootstrap starting.
I20260929 15:46:54.798725 12270 log.cc:824] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:46:54.800936 12272 tablet_bootstrap.cc:492] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Bootstrap starting.
I20260929 15:46:54.803365 12271 tablet_bootstrap.cc:654] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Neither blocks nor log segments found. Creating new log.
I20260929 15:46:54.804157 12272 tablet_bootstrap.cc:654] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Neither blocks nor log segments found. Creating new log.
I20260929 15:46:54.805516 12271 log.cc:824] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Log is configured to *not* fsync() on all Append() calls
I20260929 15:46:54.805892 12272 log.cc:824] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Log is configured to *not* fsync() on all Append() calls
I20260929 15:46:54.808986 12270 tablet_bootstrap.cc:492] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: No bootstrap required, opened a new log
I20260929 15:46:54.809425 12270 ts_tablet_manager.cc:1397] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20260929 15:46:54.810667 12272 tablet_bootstrap.cc:492] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: No bootstrap required, opened a new log
I20260929 15:46:54.810968 12272 ts_tablet_manager.cc:1397] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 15:46:54.815407 12271 tablet_bootstrap.cc:492] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: No bootstrap required, opened a new log
I20260929 15:46:54.815783 12271 ts_tablet_manager.cc:1397] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260929 15:46:54.819849 12272 raft_consensus.cc:348] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.820298 12272 raft_consensus.cc:374] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:46:54.820410 12272 raft_consensus.cc:729] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c27632d6f3c4c438e24a286c8aaf7bf, State: Initialized, Role: FOLLOWER
I20260929 15:46:54.821072 12272 consensus_queue.cc:260] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.822242 12117 heartbeater.cc:503] Master 127.0.97.62:40303 was elected leader, sending a full tablet report...
I20260929 15:46:54.822012 12270 raft_consensus.cc:348] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.822489 12270 raft_consensus.cc:374] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:46:54.822587 12270 raft_consensus.cc:729] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a394d193d6ce4630a5e51d20f24e9de8, State: Initialized, Role: FOLLOWER
I20260929 15:46:54.823647 12270 consensus_queue.cc:260] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.824430 12272 ts_tablet_manager.cc:1428] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260929 15:46:54.829026 12271 raft_consensus.cc:348] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.829829 12271 raft_consensus.cc:374] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:46:54.829996 12271 raft_consensus.cc:729] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ac403501d94e69b6d5fbb9bb910919, State: Initialized, Role: FOLLOWER
I20260929 15:46:54.831458 12251 heartbeater.cc:503] Master 127.0.97.62:40303 was elected leader, sending a full tablet report...
I20260929 15:46:54.831321 12271 consensus_queue.cc:260] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.833177 12270 ts_tablet_manager.cc:1428] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Time spent starting tablet: real 0.024s user 0.011s sys 0.012s
I20260929 15:46:54.833657 12271 ts_tablet_manager.cc:1428] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Time spent starting tablet: real 0.018s user 0.017s sys 0.002s
W20260929 15:46:54.911641 12252 tablet.cc:2369] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:46:54.922421 12276 raft_consensus.cc:482] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:46:54.922816 12276 raft_consensus.cc:504] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.924739 12276 leader_election.cc:288] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a394d193d6ce4630a5e51d20f24e9de8 (127.0.97.3:40839), 86ac403501d94e69b6d5fbb9bb910919 (127.0.97.1:40881)
I20260929 15:46:54.936757 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.926075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37179 (local address 127.0.97.1:40881)
0929 15:46:54.926375 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.926384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.929765 (+ 3381us) server_negotiation.cc:407] Connection header received
0929 15:46:54.929985 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.929989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.930049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.930152 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:46:54.930912 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.932087 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.934730 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.935573 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.935781 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.935899 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.936343 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.936478 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 15:46:54.936918 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.925425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.97.2:37179)
0929 15:46:54.929665 (+ 4240us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:54.929680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:54.929857 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:54.930178 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:54.930182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:54.930208 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:46:54.930702 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.930723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.932271 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.932278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.934505 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.934524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.935869 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.935876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.936053 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.936080 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.936314 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.936462 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4105,"mutex_wait_us":3,"thread_start_us":3930,"threads_started":1}
I20260929 15:46:54.938084 11929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b45f19cbd84d99913d6b0a9e5fbf64" candidate_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ac403501d94e69b6d5fbb9bb910919" is_pre_election: true
I20260929 15:46:54.938692 11929 raft_consensus.cc:2454] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c27632d6f3c4c438e24a286c8aaf7bf in term 0.
I20260929 15:46:54.939553 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.925349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58843 (local address 127.0.97.3:40839)
0929 15:46:54.925913 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:54.925931 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:54.929689 (+ 3758us) server_negotiation.cc:407] Connection header received
0929 15:46:54.929921 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:54.929926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:54.930020 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:54.930137 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:46:54.931036 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.932087 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.937281 (+ 5194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.938523 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.938663 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.938766 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:54.939136 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 15:46:54.939297 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260929 15:46:54.939491 12001 leader_election.cc:302] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c27632d6f3c4c438e24a286c8aaf7bf, 86ac403501d94e69b6d5fbb9bb910919; no voters:
I20260929 15:46:54.939639 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:54.925088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40839 (local address 127.0.97.2:58843)
0929 15:46:54.929521 (+ 4433us) negotiation.cc:107] Waiting for socket to connect
0929 15:46:54.929550 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:46:54.929757 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:46:54.930307 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:46:54.930312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:46:54.930355 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:46:54.930820 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.930831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.934709 (+ 3878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.934715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.937087 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:46:54.937102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:46:54.938762 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:46:54.938771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:46:54.938952 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:54.938986 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:46:54.939090 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:46:54.939259 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4220,"thread_start_us":3318,"threads_started":1}
I20260929 15:46:54.940256 12276 raft_consensus.cc:2781] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:46:54.940407 12276 raft_consensus.cc:482] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:46:54.940498 12276 raft_consensus.cc:3037] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:46:54.940950 12206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b45f19cbd84d99913d6b0a9e5fbf64" candidate_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a394d193d6ce4630a5e51d20f24e9de8" is_pre_election: true
I20260929 15:46:54.941995 12206 raft_consensus.cc:2454] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c27632d6f3c4c438e24a286c8aaf7bf in term 0.
I20260929 15:46:54.944105 12276 raft_consensus.cc:504] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.944941 12276 leader_election.cc:288] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [CANDIDATE]: Term 1 election: Requested vote from peers a394d193d6ce4630a5e51d20f24e9de8 (127.0.97.3:40839), 86ac403501d94e69b6d5fbb9bb910919 (127.0.97.1:40881)
I20260929 15:46:54.945567 11929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b45f19cbd84d99913d6b0a9e5fbf64" candidate_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ac403501d94e69b6d5fbb9bb910919"
I20260929 15:46:54.945758 12206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b45f19cbd84d99913d6b0a9e5fbf64" candidate_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a394d193d6ce4630a5e51d20f24e9de8"
I20260929 15:46:54.945822 11929 raft_consensus.cc:3037] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:46:54.945986 12206 raft_consensus.cc:3037] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:46:54.949328 12206 raft_consensus.cc:2454] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c27632d6f3c4c438e24a286c8aaf7bf in term 1.
I20260929 15:46:54.949328 11929 raft_consensus.cc:2454] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c27632d6f3c4c438e24a286c8aaf7bf in term 1.
I20260929 15:46:54.949895 12001 leader_election.cc:302] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c27632d6f3c4c438e24a286c8aaf7bf, 86ac403501d94e69b6d5fbb9bb910919; no voters:
I20260929 15:46:54.950469 12276 raft_consensus.cc:2781] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:46:54.951150 12276 raft_consensus.cc:686] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 1 LEADER]: Becoming Leader. State: Replica: 6c27632d6f3c4c438e24a286c8aaf7bf, State: Running, Role: LEADER
I20260929 15:46:54.951710 12276 consensus_queue.cc:237] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } }
I20260929 15:46:54.956856 11761 catalog_manager.cc:5495] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c27632d6f3c4c438e24a286c8aaf7bf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c27632d6f3c4c438e24a286c8aaf7bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35287 } health_report { overall_health: HEALTHY } } }
W20260929 15:46:54.986826 12118 tablet.cc:2369] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:46:55.020834 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:55.014870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.97.2:35287)
0929 15:46:55.015101 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:55.015109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:55.015133 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:46:55.015241 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:55.015245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:55.015339 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:55.015502 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:46:55.015894 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.016672 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.018525 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.018942 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.019100 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:55.019631 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:46:55.020150 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:46:55.020396 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:55.020453 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 15:46:55.020540 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:46:55.032465 11929 raft_consensus.cc:1261] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 1 FOLLOWER]: Refusing update from remote peer 6c27632d6f3c4c438e24a286c8aaf7bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:46:55.032465 12206 raft_consensus.cc:1261] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 1 FOLLOWER]: Refusing update from remote peer 6c27632d6f3c4c438e24a286c8aaf7bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:46:55.033767 12276 consensus_queue.cc:1035] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ac403501d94e69b6d5fbb9bb910919" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:46:55.034162 12281 consensus_queue.cc:1035] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a394d193d6ce4630a5e51d20f24e9de8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:46:55.048221 12285 mvcc.cc:205] Tried to move back new op lower bound from 7334694154360598528 to 7334694154046611456. Current Snapshot: MvccSnapshot[applied={T|T < 7334694154360598528}]
I20260929 15:46:55.054733 12284 mvcc.cc:205] Tried to move back new op lower bound from 7334694154360598528 to 7334694154046611456. Current Snapshot: MvccSnapshot[applied={T|T < 7334694154360598528 or (T in {7334694154360598528})}]
I20260929 15:46:55.060600 12286 mvcc.cc:205] Tried to move back new op lower bound from 7334694154360598528 to 7334694154046611456. Current Snapshot: MvccSnapshot[applied={T|T < 7334694154360598528}]
I20260929 15:46:55.074455 11760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:46:55.074817 11760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:46:55.080479 11760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b33a358921e440d5bf19ee55a6ad0cb4: Sending DeleteTablet for 3 replicas of tablet 31b45f19cbd84d99913d6b0a9e5fbf64
I20260929 15:46:55.082146 11909 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b45f19cbd84d99913d6b0a9e5fbf64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:46:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41966
I20260929 15:46:55.082427 12186 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b45f19cbd84d99913d6b0a9e5fbf64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:46:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48888
I20260929 15:46:55.082863 12052 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b45f19cbd84d99913d6b0a9e5fbf64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:46:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37368
I20260929 15:46:55.086580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 11841
I20260929 15:46:55.093659 12295 tablet_replica.cc:330] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: stopping tablet replica
I20260929 15:46:55.095954 12295 raft_consensus.cc:2229] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:46:55.097184 12295 raft_consensus.cc:2258] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:46:55.102908 12295 ts_tablet_manager.cc:1905] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:46:55.104982 12294 tablet_replica.cc:330] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: stopping tablet replica
I20260929 15:46:55.106056 12294 raft_consensus.cc:2229] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:46:55.107297 12294 raft_consensus.cc:2258] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:46:55.108059 12293 tablet_replica.cc:330] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: stopping tablet replica
I20260929 15:46:55.108626 12293 raft_consensus.cc:2229] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:46:55.109827 12293 raft_consensus.cc:2258] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:46:55.110404 12294 ts_tablet_manager.cc:1905] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:46:55.113559 12295 ts_tablet_manager.cc:1918] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:46:55.113813 12295 log.cc:1196] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/31b45f19cbd84d99913d6b0a9e5fbf64
I20260929 15:46:55.114472 12295 ts_tablet_manager.cc:1939] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 6c27632d6f3c4c438e24a286c8aaf7bf: Deleting consensus metadata
Sep 29 15:46:55 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:46:55.116786 11748 catalog_manager.cc:4841] TS 6c27632d6f3c4c438e24a286c8aaf7bf (127.0.97.2:35287): tablet 31b45f19cbd84d99913d6b0a9e5fbf64 (table test-table [id=ad3019572af64e7c8ead893d2d7130cb]) successfully deleted
I20260929 15:46:55.117724 12293 ts_tablet_manager.cc:1905] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:46:55.119599 12294 ts_tablet_manager.cc:1918] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:46:55.119907 12294 log.cc:1196] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/31b45f19cbd84d99913d6b0a9e5fbf64
I20260929 15:46:55.120603 12294 ts_tablet_manager.cc:1939] T 31b45f19cbd84d99913d6b0a9e5fbf64 P a394d193d6ce4630a5e51d20f24e9de8: Deleting consensus metadata
I20260929 15:46:55.122823 11745 catalog_manager.cc:4841] TS a394d193d6ce4630a5e51d20f24e9de8 (127.0.97.3:40839): tablet 31b45f19cbd84d99913d6b0a9e5fbf64 (table test-table [id=ad3019572af64e7c8ead893d2d7130cb]) successfully deleted
I20260929 15:46:55.126279 12293 ts_tablet_manager.cc:1918] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:46:55.126515 12293 log.cc:1196] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/31b45f19cbd84d99913d6b0a9e5fbf64
I20260929 15:46:55.127089 12293 ts_tablet_manager.cc:1939] T 31b45f19cbd84d99913d6b0a9e5fbf64 P 86ac403501d94e69b6d5fbb9bb910919: Deleting consensus metadata
I20260929 15:46:55.127756 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:55.088413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.0.97.1:40881)
0929 15:46:55.088659 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:55.088668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:55.101112 (+ 12444us) server_negotiation.cc:407] Connection header received
0929 15:46:55.101667 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:55.101674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:55.101798 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:55.101880 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:55.104147 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.105529 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.110055 (+ 4526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.110735 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.110850 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:55.118242 (+ 7392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:55.118258 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:55.118277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:55.118331 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:55.123179 (+ 4848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:55.124223 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:55.124233 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:55.124236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:55.124382 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:55.125067 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:55.125071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:55.125074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:55.125835 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:55.126388 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:55.126926 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 15:46:55.127210 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:46:55.128480 11969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:46:55.129245 11746 catalog_manager.cc:4841] TS 86ac403501d94e69b6d5fbb9bb910919 (127.0.97.1:40881): tablet 31b45f19cbd84d99913d6b0a9e5fbf64 (table test-table [id=ad3019572af64e7c8ead893d2d7130cb]) successfully deleted
I20260929 15:46:55.227285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 11841
I20260929 15:46:55.247898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 11980
Sep 29 15:46:55 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:46:55.268898 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:55.248823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.97.2:35287)
0929 15:46:55.248998 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:55.249002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:55.249020 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:46:55.249229 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:55.249233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:55.249295 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:55.249378 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:55.251117 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.251991 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.253970 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.254516 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.254632 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:55.260880 (+ 6248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:55.260898 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:55.260934 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:55.260985 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:55.264684 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:55.265868 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:55.265877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:55.265879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:55.266050 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:55.267117 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:55.267122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:55.267124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:55.267602 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:55.267905 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:55.268355 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 15:46:55.268558 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:46:55.269644 12112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:46:55.398118 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 11980
I20260929 15:46:55.417728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 12123
Sep 29 15:46:55 dist-test-slave-7491 krb5kdc[11285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:46:55.435025 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:46:55.418586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.97.3:40839)
0929 15:46:55.418768 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:46:55.418772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:46:55.418843 (+ 71us) server_negotiation.cc:407] Connection header received
0929 15:46:55.418984 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:46:55.418988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:46:55.419059 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:46:55.419168 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:46:55.420562 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.421289 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.422882 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:46:55.423527 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:46:55.423649 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:46:55.428467 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:46:55.428484 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:46:55.428506 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:46:55.428557 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:46:55.431726 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:55.432791 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:55.432798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:55.432801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:55.432965 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:46:55.433427 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:46:55.433431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:46:55.433433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:46:55.433847 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:46:55.434078 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 15:46:55.434446 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:46:55.434622 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:46:55.435822 12246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:46:55.530964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 12123
I20260929 15:46:55.547977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 11729
I20260929 15:46:55.549331 11791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:46:55.673779 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 11729
I20260929 15:46:55.697803 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 15:46:56.044802 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 15:46:56.045082 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 15:46:56.392534 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 15:46:56.393 UTC [11315] LOG: received smart shutdown request
2026-09-29 15:46:56.396 UTC [11315] LOG: background worker "logical replication launcher" (PID 11324) exited with exit code 1
2026-09-29 15:46:56.396 UTC [11319] LOG: shutting down
2026-09-29 15:46:56.575 UTC [11315] LOG: database system is shut down
2026-09-29T15:46:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (59071 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:46:56 dist-test-slave-7491 krb5kdc[12309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:46:56 dist-test-slave-7491 krb5kdc[12309](info): set up 2 sockets
Sep 29 15:46:56 dist-test-slave-7491 krb5kdc[12309](info): commencing operation
krb5kdc: starting...
W20260929 15:46:57.844928 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:46:57 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:46:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 15:46:58.185168 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQOquZz/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/postgres -l logfile start
2026-09-29 15:47:00.828 UTC [12339] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:47:00.828 UTC [12339] LOG: listening on IPv4 address "127.0.97.20", port 54385
2026-09-29 15:47:00.831 UTC [12339] LOG: listening on Unix socket "/tmp/.s.PGSQL.54385"
2026-09-29 15:47:00.854 UTC [12342] LOG: database system was shut down at 2026-09-29 15:46:59 UTC
2026-09-29 15:47:00.858 UTC [12339] LOG: database system is ready to accept connections
I20260929 15:47:01.860399 388 mini_postgres.cc:96] Postgres bound to 54385
2026-09-29 15:47:01.877 UTC [12350] FATAL: database "slave" does not exist
127.0.97.20:54385 - accepting connections
I20260929 15:47:01.878507 388 mini_ranger.cc:161] Starting Ranger...
I20260929 15:47:01.906481 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:47:02.406668 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:47:02.406810 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-admin
2026-09-29 15:47:02,705 [I] DB FLAVOR :POSTGRES
2026-09-29 15:47:02,706 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:47:02,706 [I] Checking connection..
2026-09-29 15:47:02,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:47:02,967 [I] Checking connection passed.
2026-09-29 15:47:02,968 [I] --------- Verifying version history table ---------
2026-09-29 15:47:02,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:47:03,232 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:47:03,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:47:03,492 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:47:03,492 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:47:03,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:47:03.752 UTC [12414] WARNING: there is no transaction in progress
2026-09-29 15:47:03,758 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:47:03,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:47:04,020 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:47:04,020 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:47:04,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:47:04,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:47:04,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7491.c.gcp-upstream.internal','N') ;"
2026-09-29 15:47:04,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:47:04,998 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:47:04,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:47:05,247 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:47:05,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:47:05,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:47:05,786 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:47:05,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:47:06.770 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.788 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.805 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.821 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.837 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.855 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.922 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.929 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.944 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.956 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.966 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.980 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.990 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:06.997 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:07.004 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:07.011 UTC [12522] WARNING: there is no transaction in progress
2026-09-29 15:47:07,640 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:47:07,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:47:07,902 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:47:07,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:47:08,159 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:47:08,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:47:08,407 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:47:08,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:47:08,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:47:08,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7491.c.gcp-upstream.internal';"
2026-09-29 15:47:09,194 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:47:09,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:47:09,460 [I] DB_PATCHES have already been applied
I20260929 15:47:09.466308 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQOquZz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:47:09.466410 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 15:47:09.481477 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:55213
Sep 29, 2026 3:47:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:47:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:47:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:47:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:47:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:47:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:47:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:47:10.623628 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.745s user 0.000s sys 0.041s
Sep 29, 2026 3:47:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:47:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:47:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:47:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:47:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 15:47:33.989130 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 15:47:34 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696854, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 15:47:34.182551 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 15:47:34.206054 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 15:47:35.143280 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 15:47:35.143394 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms
I20260929 15:47:35.143438 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54385
2026-09-29 15:47:35,517 [I] DB FLAVOR :POSTGRES
2026-09-29 15:47:35,518 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:47:35,518 [I] Checking connection
2026-09-29 15:47:35,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 15:47:35,791 [I] connection success
2026-09-29 15:47:35,791 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 15:47:35,791 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 15:47:35,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 15:47:36,062 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 15:47:36,062 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:47:36,062 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 15:47:36,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 15:47:36,368 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 15:47:36.984755 388 mini_ranger_kms.cc:310] Created kms service
I20260929 15:47:37.215750 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 15:47:37.272075 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 15:47:37 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696854, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:37.719631 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 15:47:37.720109 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQOquZz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQOquZz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 15:47:37.720227 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 15:47:37.738224 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12703
I20260929 15:47:37.738373 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:52493
Sep 29, 2026 3:47:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:47:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:47:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:47:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:47:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:47:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 15:47:40.080349 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.898s user 0.064s sys 0.326s
I20260929 15:47:40.080814 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 15:47:40.080997 388 mini_ranger_kms.cc:401] 127.0.97.20:52493/kms/v1/keys
Sep 29 15:47:40 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790696860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 15:47:40 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:47:41 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696860, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 15:47:43 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:47:43.883260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43063
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43063
--ranger_config_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:47:44.250224 12753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:44.250550 12753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:44.250787 12753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:44.258603 12753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:47:44.258736 12753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:44.258782 12753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:44.258831 12753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:47:44.258872 12753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:47:44.269959 12753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:52493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43063
--ranger_config_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:44.271260 12753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:44.272554 12753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:44.285589 12758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:44.285529 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:44.285634 12761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:44.286428 12753 server_base.cc:1034] running on GCE node
I20260929 15:47:44.287376 12753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:44.289131 12753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:44.290378 12753 hybrid_clock.cc:648] HybridClock initialized: now 1790696864290325 us; error 37 us; skew 500 ppm
I20260929 15:47:44.290799 12753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:44 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:44.301131 12753 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:47:44.303056 12753 webserver.cc:460] Webserver started at http://127.0.97.62:33713/ using document root <none> and password file <none>
I20260929 15:47:44.303810 12753 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:44.303966 12753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:44.304394 12753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:47:44 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:44.378991 12753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "043492bbb8604477852ad37a4ef15d2c"
format_stamp: "Formatted at 2026-09-29 15:47:44 on dist-test-slave-7491"
server_key: "b33126c0a8e32d3908badaf17d8b18bd"
server_key_iv: "a7afc7db50f7f39304c552d77da3478a"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:44.380203 12753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "043492bbb8604477852ad37a4ef15d2c"
format_stamp: "Formatted at 2026-09-29 15:47:44 on dist-test-slave-7491"
server_key: "b33126c0a8e32d3908badaf17d8b18bd"
server_key_iv: "a7afc7db50f7f39304c552d77da3478a"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:44.387943 12753 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 15:47:44.409503 12768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:44.411445 12753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:47:44.411708 12753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "043492bbb8604477852ad37a4ef15d2c"
format_stamp: "Formatted at 2026-09-29 15:47:44 on dist-test-slave-7491"
server_key: "b33126c0a8e32d3908badaf17d8b18bd"
server_key_iv: "a7afc7db50f7f39304c552d77da3478a"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:44.411999 12753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:44.437433 12753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:44.442253 12753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:44.442607 12753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:44.466497 12753 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43063
I20260929 15:47:44.466514 12819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43063 every 8 connection(s)
I20260929 15:47:44.468089 12753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:47:44.472039 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 12753
I20260929 15:47:44.472312 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:47:44.472790 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:44.473124 388 external_mini_cluster.cc:1428] Setting key 991b0cea82c907132290f0db57a13297
I20260929 15:47:44.493096 12820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Bootstrap starting.
I20260929 15:47:44.497592 12820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:44.499055 12820 log.cc:824] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:44.503203 12820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: No bootstrap required, opened a new log
Sep 29 15:47:44 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:44.514962 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:44.475501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.97.62:43063)
0929 15:47:44.488908 (+ 13407us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:44.488933 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:44.489594 (+ 661us) server_negotiation.cc:407] Connection header received
0929 15:47:44.492502 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:44.492544 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:44.493205 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:44.493724 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:44.495552 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:44.497289 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:44.499394 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:44.500267 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:44.500450 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:44.505245 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:44.505257 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:44.505279 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:44.505337 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:44.509741 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:44.510647 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:44.510659 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:44.510669 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:44.510889 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:44.511270 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:44.511276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:44.511279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:44.512123 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:44.512470 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:44.512945 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 15:47:44.513467 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13107,"thread_start_us":12729,"threads_started":1}
I20260929 15:47:44.515120 12820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:44.515600 12820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:44.515739 12820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043492bbb8604477852ad37a4ef15d2c, State: Initialized, Role: FOLLOWER
I20260929 15:47:44.516456 12820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:44.516857 12820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:47:44.517006 12820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:47:44.517210 12820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:44.520402 12820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:44.521212 12820 leader_election.cc:302] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043492bbb8604477852ad37a4ef15d2c; no voters:
I20260929 15:47:44.521981 12820 leader_election.cc:288] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:47:44.522401 12825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:47:44.524355 12825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 LEADER]: Becoming Leader. State: Replica: 043492bbb8604477852ad37a4ef15d2c, State: Running, Role: LEADER
I20260929 15:47:44.525036 12825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:44.525722 12820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:47:44.527904 12827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043492bbb8604477852ad37a4ef15d2c. Latest consensus state: current_term: 1 leader_uuid: "043492bbb8604477852ad37a4ef15d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } } }
I20260929 15:47:44.528350 12827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:44.528964 12826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043492bbb8604477852ad37a4ef15d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } } }
I20260929 15:47:44.529295 12826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:44.533638 12830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:47:44.544551 12830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:47:44.545367 12820 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 15:47:44.564992 12830 catalog_manager.cc:1347] Generated new cluster ID: 2f6a8959c4a3489c8f8b3ba3df0bd529
I20260929 15:47:44.565157 12830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:47:44.613353 12830 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:47:44.615453 12830 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:47:44.638926 12830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Generated new TSK 0
I20260929 15:47:44.640288 12830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:47:45.344123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43063
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:47:45 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790696865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 15:47:45.771396 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:45.771944 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:45.772195 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:45.784404 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:45.784693 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:45.784978 12866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:47:45.801599 12866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:52493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:45.803683 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:45.805572 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:45.823308 12876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:45.823738 12866 server_base.cc:1034] running on GCE node
W20260929 15:47:45.824524 12879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:45.824792 12877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:45.825603 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:45.826812 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:45.828076 12866 hybrid_clock.cc:648] HybridClock initialized: now 1790696865828013 us; error 54 us; skew 500 ppm
I20260929 15:47:45.828500 12866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:45 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:45.832991 12866 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:47:45.838086 12866 webserver.cc:460] Webserver started at http://127.0.97.1:43781/ using document root <none> and password file <none>
I20260929 15:47:45.839140 12866 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:45.839322 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:45.839754 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:47:45 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:45.864204 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "5e4422b6a0e04c09a65782c4d616a4ef"
format_stamp: "Formatted at 2026-09-29 15:47:45 on dist-test-slave-7491"
server_key: "e0762d272dc8116d3c29692cb0102f70"
server_key_iv: "11ac428c25e34c8a4f56497aeac0157d"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:45.865477 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e4422b6a0e04c09a65782c4d616a4ef"
format_stamp: "Formatted at 2026-09-29 15:47:45 on dist-test-slave-7491"
server_key: "e0762d272dc8116d3c29692cb0102f70"
server_key_iv: "11ac428c25e34c8a4f56497aeac0157d"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:45.886662 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.003s sys 0.016s
I20260929 15:47:45.913416 12886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:45.915359 12866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:47:45.915650 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5e4422b6a0e04c09a65782c4d616a4ef"
format_stamp: "Formatted at 2026-09-29 15:47:45 on dist-test-slave-7491"
server_key: "e0762d272dc8116d3c29692cb0102f70"
server_key_iv: "11ac428c25e34c8a4f56497aeac0157d"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:45.916152 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:45.957486 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:45.966141 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:45.966593 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:45.974102 12866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:47:45.982502 12866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:47:45.982698 12866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:47:45.982925 12866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:47:45.983042 12866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:47:46 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:46.444698 12998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33065 every 8 connection(s)
I20260929 15:47:46.444983 12866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33065
I20260929 15:47:46.446931 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:47:46.455046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 12866
I20260929 15:47:46.455271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:47:46.456117 388 external_mini_cluster.cc:1428] Setting key ca5c070d07e23b47160343069a3a055a
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 29 15:47:46 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:46.530404 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:46.477165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57725 (local address 127.0.97.62:43063)
0929 15:47:46.502099 (+ 24934us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:46.502106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:46.502137 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:47:46.502755 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:46.502761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:46.502855 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:46.502957 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:46.512956 (+ 9999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:46.514228 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:46.517017 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:46.517689 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:46.517804 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:46.522796 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:46.522805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:46.522811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:46.522859 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:46.526574 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:46.527767 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:46.527776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:46.527780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:46.527945 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:46.528521 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:46.528526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:46.528529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:46.528829 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:46.529003 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:46.529746 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 15:47:46.529986 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24780,"thread_start_us":24963,"threads_started":1}
I20260929 15:47:46.531410 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:46.477727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43063 (local address 127.0.97.1:57725)
0929 15:47:46.497683 (+ 19956us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:46.497759 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:46.500134 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:46.503911 (+ 3777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:46.503925 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:46.504815 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:46.512583 (+ 7768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:46.512650 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:46.514524 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:46.514530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:46.516807 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:46.516830 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:46.518192 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:46.518200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:46.518531 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:46.519442 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:46.519476 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:46.522415 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:46.526803 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:46.526813 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:46.526835 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:46.527571 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:46.528080 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:46.528083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:46.528085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:46.528379 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:46.528973 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:46.528983 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:46.529351 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 15:47:46.529651 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 15:47:46.530211 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19540,"thread_start_us":19251,"threads_started":1}
I20260929 15:47:46.533272 12999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43063
I20260929 15:47:46.533629 12999 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:46.534525 12999 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:46.537165 12785 ts_manager.cc:194] Registered new tserver with Master: 5e4422b6a0e04c09a65782c4d616a4ef (127.0.97.1:33065)
I20260929 15:47:46.539114 12785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:47:46.543313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43063
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:47:46 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 15:47:47.051383 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:47.051723 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:47.051888 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:47.060350 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:47.060542 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:47.060745 13005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:47:47.072485 13005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:52493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:47.073937 13005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:47.075347 13005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:47.092455 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:47.092886 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:47.093161 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:47.094275 13005 server_base.cc:1034] running on GCE node
I20260929 15:47:47.095108 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:47.096330 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:47.097731 13005 hybrid_clock.cc:648] HybridClock initialized: now 1790696867097550 us; error 57 us; skew 500 ppm
I20260929 15:47:47.098397 13005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:47.102700 13005 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:47:47.105074 13005 webserver.cc:460] Webserver started at http://127.0.97.2:34701/ using document root <none> and password file <none>
I20260929 15:47:47.106227 13005 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:47.106479 13005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:47.106873 13005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:47.125672 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "e7f2ba4a11344aeea65655944e60eaa0"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "dfbc5120655afa3176e46f21ca6445c9"
server_key_iv: "c039d59a0b1a5da4fa0deaf0b0655029"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:47.126773 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e7f2ba4a11344aeea65655944e60eaa0"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "dfbc5120655afa3176e46f21ca6445c9"
server_key_iv: "c039d59a0b1a5da4fa0deaf0b0655029"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:47.133456 13005 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:47:47.152381 13029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:47.154398 13005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:47:47.154687 13005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "e7f2ba4a11344aeea65655944e60eaa0"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "dfbc5120655afa3176e46f21ca6445c9"
server_key_iv: "c039d59a0b1a5da4fa0deaf0b0655029"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:47.155006 13005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:47.180140 13005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:47.185482 13005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:47.185822 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:47.187021 13005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:47:47.188714 13005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:47:47.188794 13005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:47.188944 13005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:47:47.188998 13005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:47.231997 13005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37821
I20260929 15:47:47.232045 13141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37821 every 8 connection(s)
I20260929 15:47:47.234295 13005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:47:47.237538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13005
I20260929 15:47:47.237732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:47:47.238229 388 external_mini_cluster.cc:1428] Setting key f5967b0a4f70d01b5cce450be04e6fe3
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:47.258355 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:47.238625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59363 (local address 127.0.97.62:43063)
0929 15:47:47.239441 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:47.239448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:47.240480 (+ 1032us) server_negotiation.cc:407] Connection header received
0929 15:47:47.242446 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:47.242452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:47.242548 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:47.243039 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:47.244721 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.245479 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.247136 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.247556 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.247727 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:47.252050 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:47.252065 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:47.252068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:47.252112 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:47.254946 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:47.255811 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:47.255818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:47.255822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:47.255967 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:47.256485 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:47.256490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:47.256493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:47.256790 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:47.256959 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:47.257722 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 15:47:47.257977 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":551,"threads_started":1}
I20260929 15:47:47.259552 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:47.239352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43063 (local address 127.0.97.2:59363)
0929 15:47:47.240325 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:47.240374 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:47.242134 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:47.243332 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:47.243343 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:47.243810 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:47.244518 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:47.244536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.245658 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.245665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:47.246960 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:47.246974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.247857 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.247864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:47.248105 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:47.248934 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:47.248958 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:47.251766 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:47.255137 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:47.255150 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:47.255167 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:47.255666 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:47.256092 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:47.256097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:47.256101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:47.256357 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:47.256931 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:47.256940 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:47.257310 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 15:47:47.257595 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 15:47:47.258157 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":413,"threads_started":1}
I20260929 15:47:47.261461 13142 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43063
I20260929 15:47:47.262167 13142 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:47.263109 13142 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:47.264892 12785 ts_manager.cc:194] Registered new tserver with Master: e7f2ba4a11344aeea65655944e60eaa0 (127.0.97.2:37821)
I20260929 15:47:47.265755 12785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59363
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:47:47.303257 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43063
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:47:47.542795 12999 heartbeater.cc:503] Master 127.0.97.62:43063 was elected leader, sending a full tablet report...
W20260929 15:47:47.682765 13148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:47.683166 13148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:47.683341 13148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:47.692538 13148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:47.692715 13148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:47.692893 13148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:47:47.704638 13148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:52493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:47.706321 13148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:47.707787 13148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:47.722963 13156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:47.723061 13154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:47.723095 13153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:47.723702 13148 server_base.cc:1034] running on GCE node
I20260929 15:47:47.724684 13148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:47.725821 13148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:47.727095 13148 hybrid_clock.cc:648] HybridClock initialized: now 1790696867727010 us; error 99 us; skew 500 ppm
I20260929 15:47:47.727479 13148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:47.731599 13148 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:47:47.733530 13148 webserver.cc:460] Webserver started at http://127.0.97.3:43437/ using document root <none> and password file <none>
I20260929 15:47:47.734308 13148 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:47.734469 13148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:47.734812 13148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:47.752068 13148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "f93767679c0f4da3b522eaa4dd12f4cc"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "c3ac3725bce31c2748c9ee32dfdfa2f0"
server_key_iv: "321cdd7c2de741b656d21f8a30cbed2b"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:47.753072 13148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f93767679c0f4da3b522eaa4dd12f4cc"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "c3ac3725bce31c2748c9ee32dfdfa2f0"
server_key_iv: "321cdd7c2de741b656d21f8a30cbed2b"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:47.759075 13148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:47:47.778810 13163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:47.781636 13148 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:47:47.781932 13148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "f93767679c0f4da3b522eaa4dd12f4cc"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "c3ac3725bce31c2748c9ee32dfdfa2f0"
server_key_iv: "321cdd7c2de741b656d21f8a30cbed2b"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:47.782279 13148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:47.820519 13148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:47.825454 13148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:47.825790 13148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:47.826879 13148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:47:47.828639 13148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:47:47.828730 13148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:47.828902 13148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:47:47.828958 13148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:47.866880 13148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32895
I20260929 15:47:47.866938 13275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32895 every 8 connection(s)
I20260929 15:47:47.868280 13148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:47:47.874718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13148
I20260929 15:47:47.874934 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:47:47.875396 388 external_mini_cluster.cc:1428] Setting key e9861d0f96c9360d62e3c418f5f588da
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:47.891067 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:47.871307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38339 (local address 127.0.97.62:43063)
0929 15:47:47.871865 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:47.871870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:47.872677 (+ 807us) server_negotiation.cc:407] Connection header received
0929 15:47:47.874634 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:47.874640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:47.874756 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:47.874887 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:47.876764 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.877668 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.879530 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.880093 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.880299 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:47.883849 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:47.883855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:47.883858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:47.883893 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:47.887463 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:47.888441 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:47.888449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:47.888457 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:47.888594 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:47.889174 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:47.889184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:47.889186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:47.889487 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:47.889666 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:47.890393 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 15:47:47.890661 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":281,"threads_started":1}
I20260929 15:47:47.892238 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:47.871664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43063 (local address 127.0.97.3:38339)
0929 15:47:47.872494 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:47.872550 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:47.874301 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:47.875168 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:47.875178 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:47.875709 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:47.876555 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:47.876578 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.877845 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.877849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:47.879321 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:47.879335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.880392 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.880397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:47.880626 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:47.881356 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:47.881380 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:47.883566 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:47.887670 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:47.887689 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:47.887720 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:47.888293 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:47.888720 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:47.888725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:47.888729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:47.889020 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:47.889638 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:47.889646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:47.890005 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 15:47:47.890289 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 15:47:47.890844 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":302,"threads_started":1}
I20260929 15:47:47.894100 13276 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43063
I20260929 15:47:47.894479 13276 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:47.895268 13276 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:47.897233 12785 ts_manager.cc:194] Registered new tserver with Master: f93767679c0f4da3b522eaa4dd12f4cc (127.0.97.3:32895)
I20260929 15:47:47.898124 12785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38339
I20260929 15:47:47.905267 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:47:47.907143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 12866
Sep 29 15:47:47 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:47:47.927788 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:47.908450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.0.97.1:33065)
0929 15:47:47.909128 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:47.909135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:47.909169 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:47:47.909229 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:47.909236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:47.909445 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:47.909575 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:47.911055 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.912129 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.914267 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:47.914873 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:47.914993 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:47.920553 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:47.920570 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:47.920595 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:47.920652 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:47.924247 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:47.925283 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:47.925292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:47.925294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:47.925478 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:47.925965 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:47.925969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:47.925971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:47.926352 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:47.926737 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:47.927091 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 15:47:47.927381 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":376,"threads_started":1}
I20260929 15:47:47.929304 12994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:48.016407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 12866
I20260929 15:47:48.032404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13005
Sep 29 15:47:48 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:47:48.054518 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:48.033346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.0.97.2:37821)
0929 15:47:48.033837 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:48.033844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:48.033868 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:47:48.033940 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:48.033946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:48.034164 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:48.034344 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:48.035599 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:48.036732 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:48.038306 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:48.038962 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:48.039111 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:48.048445 (+ 9334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:48.048464 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:48.048481 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:48.048524 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:48.051474 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:48.052303 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:48.052311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:48.052313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:48.052456 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:48.052880 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:48.052885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:48.052888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:48.053252 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:48.053582 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:48.053883 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 15:47:48.054151 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260929 15:47:48.055943 13137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:48.142390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13005
I20260929 15:47:48.159525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13148
Sep 29 15:47:48 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:47:48.176944 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:48.160635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.0.97.3:32895)
0929 15:47:48.161276 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:48.161284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:48.161314 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:47:48.161375 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:48.161381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:48.161625 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:48.161797 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:48.163138 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:48.164156 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:48.165497 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:48.166076 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:48.166249 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:48.170741 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:48.170756 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:48.170772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:48.170812 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:48.173952 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:48.174801 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:48.174815 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:48.174820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:48.174999 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:48.175408 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:48.175412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:48.175414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:48.175804 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:48.176058 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:48.176441 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 15:47:48.176659 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":321,"threads_started":1}
I20260929 15:47:48.178350 13271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:48.262436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13148
I20260929 15:47:48.278455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 12753
I20260929 15:47:48.279858 12815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:48.388361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 12753
I20260929 15:47:48.410284 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43063
--webserver_interface=127.0.97.62
--webserver_port=33713
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43063
--ranger_config_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:47:48.758174 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:48.758574 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:48.758734 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:48.766527 13288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:47:48.766660 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:48.766709 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:48.766804 13288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:47:48.766880 13288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:47:48.778499 13288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43063
--ranger_config_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:48.780138 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:48.781745 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:48.794479 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:48.794468 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:48.794692 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:48.795485 13288 server_base.cc:1034] running on GCE node
I20260929 15:47:48.796487 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:48.798090 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:48.799384 13288 hybrid_clock.cc:648] HybridClock initialized: now 1790696868799331 us; error 72 us; skew 500 ppm
I20260929 15:47:48.799782 13288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:48 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:48.803102 13288 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:47:48.805292 13288 webserver.cc:460] Webserver started at http://127.0.97.62:33713/ using document root <none> and password file <none>
I20260929 15:47:48.806277 13288 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:48.806475 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:47:48 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:48.879890 13288 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 15:47:48.883188 13304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:48.884835 13288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:47:48.885084 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "043492bbb8604477852ad37a4ef15d2c"
format_stamp: "Formatted at 2026-09-29 15:47:44 on dist-test-slave-7491"
server_key: "b33126c0a8e32d3908badaf17d8b18bd"
server_key_iv: "a7afc7db50f7f39304c552d77da3478a"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:48.886405 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:48.899024 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:48.903499 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:48.903771 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:48.923247 13288 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43063
I20260929 15:47:48.923266 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43063 every 8 connection(s)
I20260929 15:47:48.924806 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:47:48.926357 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13288
I20260929 15:47:48.927404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33065
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43063
--webserver_port=43781
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:47:48.935280 13356 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:47:48.938925 13356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Bootstrap starting.
I20260929 15:47:48.959028 13356 log.cc:824] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:48.968425 13356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:47:48.969005 13356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Bootstrap complete.
I20260929 15:47:48.978461 13356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:48.979357 13356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043492bbb8604477852ad37a4ef15d2c, State: Initialized, Role: FOLLOWER
I20260929 15:47:48.980020 13356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:48.980309 13356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:47:48.980453 13356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:47:48.980654 13356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:47:48.985033 13356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:48.985596 13356 leader_election.cc:302] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043492bbb8604477852ad37a4ef15d2c; no voters:
I20260929 15:47:48.986321 13356 leader_election.cc:288] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:47:48.986480 13360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:47:48.987540 13360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [term 2 LEADER]: Becoming Leader. State: Replica: 043492bbb8604477852ad37a4ef15d2c, State: Running, Role: LEADER
I20260929 15:47:48.988152 13360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } }
I20260929 15:47:48.988713 13356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:47:48.993496 13361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "043492bbb8604477852ad37a4ef15d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } } }
I20260929 15:47:48.994151 13361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:48.997213 13362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043492bbb8604477852ad37a4ef15d2c. Latest consensus state: current_term: 2 leader_uuid: "043492bbb8604477852ad37a4ef15d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043492bbb8604477852ad37a4ef15d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43063 } } }
I20260929 15:47:48.997654 13362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:49.000085 13366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:47:49.007936 13366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:47:49.010103 13366 catalog_manager.cc:1259] Loaded cluster ID: 2f6a8959c4a3489c8f8b3ba3df0bd529
I20260929 15:47:49.010264 13366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:47:49.015784 13366 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:47:49.017998 13366 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Loaded TSK: 0
I20260929 15:47:49.019985 13366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:47:49.022710 13356 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 15:47:49.420264 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:49.420663 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:49.420822 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:49.433470 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:49.433656 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:49.433856 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:47:49.450872 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43781
--tserver_master_addrs=127.0.97.62:43063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:49.452842 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:49.454838 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:49.505996 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:49.506237 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:49.512113 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:49.514451 13358 server_base.cc:1034] running on GCE node
I20260929 15:47:49.515663 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:49.523291 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:49.524842 13358 hybrid_clock.cc:648] HybridClock initialized: now 1790696869524727 us; error 121 us; skew 500 ppm
I20260929 15:47:49.525296 13358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:49 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:49.533728 13358 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:47:49.546067 13358 webserver.cc:460] Webserver started at http://127.0.97.1:43781/ using document root <none> and password file <none>
I20260929 15:47:49.547171 13358 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:49.547380 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:47:49 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:49.586450 13358 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.004s sys 0.003s
I20260929 15:47:49.598248 13405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:49.600503 13358 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.010s sys 0.000s
I20260929 15:47:49.600826 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5e4422b6a0e04c09a65782c4d616a4ef"
format_stamp: "Formatted at 2026-09-29 15:47:45 on dist-test-slave-7491"
server_key: "e0762d272dc8116d3c29692cb0102f70"
server_key_iv: "11ac428c25e34c8a4f56497aeac0157d"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:49.602409 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:49.658612 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:49.664469 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:49.664798 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:49.666174 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:47:49.669091 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:47:49.669216 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:49.669318 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:47:49.669377 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:49.736485 13358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33065
I20260929 15:47:49.736532 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33065 every 8 connection(s)
I20260929 15:47:49.738508 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:47:49.748150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13358
I20260929 15:47:49.748950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37821
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43063
--webserver_port=34701
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:47:49 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:49.779331 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:49.742227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43063 (local address 127.0.97.1:55075)
0929 15:47:49.743418 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:49.743465 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:49.745417 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:49.747007 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:49.747018 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:49.747687 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:49.748567 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:49.748596 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:49.751218 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:49.751227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:49.753670 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:49.753721 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:49.755879 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:49.755888 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:49.756227 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:49.760642 (+ 4415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:49.760684 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:49.763573 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:49.771980 (+ 8407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:49.771992 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:49.772013 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:49.773240 (+ 1227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:49.774447 (+ 1207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:49.774454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:49.774459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:49.774803 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:49.776098 (+ 1295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:49.776107 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:49.776599 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 15:47:49.776962 (+ 363us) client_negotiation.cc:240] Negotiation successful
0929 15:47:49.777616 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":744,"threads_started":1}
I20260929 15:47:49.786980 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:49.742076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55075 (local address 127.0.97.62:43063)
0929 15:47:49.743104 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:49.743125 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:49.743606 (+ 481us) server_negotiation.cc:407] Connection header received
0929 15:47:49.746055 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:49.746068 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:49.746478 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:49.746909 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:49.748863 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:49.750961 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:49.753932 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:49.755148 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:49.755406 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:49.764070 (+ 8664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:49.764085 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:49.764108 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:49.764207 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:49.771538 (+ 7331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:49.773647 (+ 2109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:49.773668 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:49.773683 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:49.774182 (+ 499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:49.774996 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:49.775002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:49.775006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:49.775774 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:49.776000 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:49.784776 (+ 8776us) server_negotiation.cc:299] Negotiation successful
0929 15:47:49.785353 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":373,"threads_started":1}
I20260929 15:47:49.794102 13518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43063
I20260929 15:47:49.794507 13518 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:49.795496 13518 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
W20260929 15:47:49.799324 13518 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43063 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:47:49.799577 13518 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:49.799872 13518 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:49.801378 13518 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:49.801749 13518 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
W20260929 15:47:49.803205 13518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:47:50.207862 13522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:50.208405 13522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:50.208616 13522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:50.221017 13522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:50.221273 13522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:50.221501 13522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:47:50.238344 13522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34701
--tserver_master_addrs=127.0.97.62:43063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:50.240406 13522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:50.242419 13522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:50.260924 13538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:50.264638 13539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:50.271860 13541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:50.272454 13522 server_base.cc:1034] running on GCE node
I20260929 15:47:50.273294 13522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:50.274617 13522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:50.275883 13522 hybrid_clock.cc:648] HybridClock initialized: now 1790696870275817 us; error 47 us; skew 500 ppm
I20260929 15:47:50.276333 13522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:50.280738 13522 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:47:50.282917 13522 webserver.cc:460] Webserver started at http://127.0.97.2:34701/ using document root <none> and password file <none>
I20260929 15:47:50.283861 13522 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:50.284014 13522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:50.319098 13522 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:47:50.327237 13549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:50.329355 13522 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260929 15:47:50.329589 13522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "e7f2ba4a11344aeea65655944e60eaa0"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "dfbc5120655afa3176e46f21ca6445c9"
server_key_iv: "c039d59a0b1a5da4fa0deaf0b0655029"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:50.331059 13522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:50.362876 13522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:50.367861 13522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:50.368242 13522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:50.369524 13522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:47:50.371536 13522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:47:50.371654 13522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:50.371767 13522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:47:50.371845 13522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:50.425448 13522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37821
I20260929 15:47:50.425493 13662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37821 every 8 connection(s)
I20260929 15:47:50.427003 13522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:47:50.435654 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13522
I20260929 15:47:50.436544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:32895
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43063
--webserver_port=43437
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:50.457259 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:50.430215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53781 (local address 127.0.97.62:43063)
0929 15:47:50.430773 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:50.430779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:50.431889 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 15:47:50.434350 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:50.434356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:50.434470 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:50.434626 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:50.436778 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:50.438039 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:50.440827 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:50.441479 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:50.441617 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:50.446604 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:50.446612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:50.446619 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:50.446671 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:50.450803 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:50.452012 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:50.452021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:50.452027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:50.452240 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:50.452823 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:50.452831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:50.452837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:50.453282 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:50.453438 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:50.456439 (+ 3001us) server_negotiation.cc:299] Negotiation successful
0929 15:47:50.456736 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":280,"threads_started":1}
I20260929 15:47:50.458109 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:50.430761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43063 (local address 127.0.97.2:53781)
0929 15:47:50.431677 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:50.431732 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:50.434026 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:50.434911 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:50.434920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:50.435521 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:50.436498 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:50.436528 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:50.438272 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:50.438280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:50.440594 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:50.440613 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:50.442062 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:50.442072 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:50.442448 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:50.443375 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:50.443411 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:50.446284 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:50.451040 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:50.451053 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:50.451076 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:50.451834 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:50.452387 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:50.452395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:50.452401 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:50.452679 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:50.453530 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:50.453542 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:50.454028 (+ 486us) client_negotiation.cc:769] Sending connection context
0929 15:47:50.454361 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 15:47:50.455019 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":390,"threads_started":1}
I20260929 15:47:50.461033 13663 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43063
I20260929 15:47:50.461391 13663 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:50.462157 13663 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:50.464360 13321 ts_manager.cc:194] Registered new tserver with Master: e7f2ba4a11344aeea65655944e60eaa0 (127.0.97.2:37821)
I20260929 15:47:50.466873 13321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53781
I20260929 15:47:50.803365 13518 heartbeater.cc:464] Registering TS with master...
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:50.803813 13518 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:50.805218 13321 ts_manager.cc:194] Registered new tserver with Master: 5e4422b6a0e04c09a65782c4d616a4ef (127.0.97.1:33065)
I20260929 15:47:50.806500 13321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55075
W20260929 15:47:50.915769 13667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:50.916234 13667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:50.916401 13667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:50.925226 13667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:50.925398 13667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:50.925570 13667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:47:50.936113 13667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:52493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:32895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43437
--tserver_master_addrs=127.0.97.62:43063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:50.937624 13667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:50.938962 13667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:50.953323 13673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:50.953832 13675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:50.954299 13672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:50.957443 13667 server_base.cc:1034] running on GCE node
I20260929 15:47:50.958863 13667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:50.960577 13667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:50.961941 13667 hybrid_clock.cc:648] HybridClock initialized: now 1790696870961874 us; error 78 us; skew 500 ppm
I20260929 15:47:50.962383 13667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:50.966959 13667 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:47:50.969174 13667 webserver.cc:460] Webserver started at http://127.0.97.3:43437/ using document root <none> and password file <none>
I20260929 15:47:50.971143 13667 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:50.971328 13667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 15:47:50 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:51.014102 13667 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.003s sys 0.007s
I20260929 15:47:51.019979 13683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:51.022146 13667 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:47:51.022459 13667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "f93767679c0f4da3b522eaa4dd12f4cc"
format_stamp: "Formatted at 2026-09-29 15:47:47 on dist-test-slave-7491"
server_key: "c3ac3725bce31c2748c9ee32dfdfa2f0"
server_key_iv: "321cdd7c2de741b656d21f8a30cbed2b"
server_key_version: "kuduclusterkey@0"
I20260929 15:47:51.023983 13667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:51.107280 13667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:51.112967 13667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:51.113294 13667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:51.114418 13667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:47:51.129537 13667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:47:51.129819 13667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:47:51.130064 13667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:47:51.130213 13667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:47:51 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 15:47:51.271263 13667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32895
I20260929 15:47:51.271314 13796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32895 every 8 connection(s)
I20260929 15:47:51.273416 13667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:47:51.273826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13667
I20260929 15:47:51.311771 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.275381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.0.97.62:43063)
0929 15:47:51.289373 (+ 13992us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.289380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.289410 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:47:51.289488 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.289494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.289585 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.289742 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:51.296952 (+ 7210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.298361 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.300593 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.301200 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.301341 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.305150 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:51.305155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:51.305159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:51.305209 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:51.308556 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.309612 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.309619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.309629 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.309759 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.310213 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.310217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.310219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.310502 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:51.310794 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.311103 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.311357 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":13843,"thread_start_us":13661,"threads_started":1}
Sep 29 15:47:51 dist-test-slave-7491 krb5kdc[12309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:51.328299 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.294126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46683 (local address 127.0.97.62:43063)
0929 15:47:51.298841 (+ 4715us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.298846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.298865 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:47:51.300647 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.300654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.300856 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.301026 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:51.303554 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.304819 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.307474 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.308281 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.308537 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.317907 (+ 9370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:51.317912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:51.317915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:51.317952 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:51.321340 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.322384 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.322392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.322396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.322534 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.322999 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.323006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.323009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.323500 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:51.323759 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.326470 (+ 2711us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.327844 (+ 1374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4549,"thread_start_us":2920,"threads_started":1}
I20260929 15:47:51.330782 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.293365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43063 (local address 127.0.97.3:46683)
0929 15:47:51.295163 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.295233 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:51.300209 (+ 4976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:51.301629 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:51.301645 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:51.302419 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:51.303322 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.303344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.305040 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.305046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.307242 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.307261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.309223 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.309233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.313333 (+ 4100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.315243 (+ 1910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:51.315268 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:51.317629 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:51.321546 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:51.321558 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:51.321579 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:51.322243 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:51.322678 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:51.322683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:51.322687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:51.322879 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:51.323915 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:51.323927 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:51.324765 (+ 838us) client_negotiation.cc:769] Sending connection context
0929 15:47:51.325132 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 15:47:51.329109 (+ 3977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"thread_start_us":1089,"threads_started":1}
I20260929 15:47:51.332754 13797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43063
I20260929 15:47:51.333127 13797 heartbeater.cc:464] Registering TS with master...
I20260929 15:47:51.334053 13797 heartbeater.cc:511] Master 127.0.97.62:43063 requested a full tablet report, sending...
I20260929 15:47:51.336010 13321 ts_manager.cc:194] Registered new tserver with Master: f93767679c0f4da3b522eaa4dd12f4cc (127.0.97.3:32895)
I20260929 15:47:51.337074 13321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46683
I20260929 15:47:51.339978 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:47:51.373024 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.355976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.0.97.62:43063)
0929 15:47:51.356172 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.356179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.356325 (+ 146us) server_negotiation.cc:407] Connection header received
0929 15:47:51.356636 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.356640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.356718 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.356840 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:51.358873 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.359984 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.362038 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.362713 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.362832 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.366662 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:51.366668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:51.366671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:51.366727 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:51.369477 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.370401 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.370408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.370412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.370567 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.371047 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.371053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.371056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.371486 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:51.371732 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.372415 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.372671 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:47:51.379240 13321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:47:51.389267 13321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:47:51.472297 13663 heartbeater.cc:503] Master 127.0.97.62:43063 was elected leader, sending a full tablet report...
I20260929 15:47:51.478582 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.406611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37821 (local address 127.0.0.1:41986)
0929 15:47:51.464693 (+ 58082us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.464718 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:51.464903 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:51.466313 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:51.466324 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:51.466384 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:47:51.466909 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.466926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.469509 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.469518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.474993 (+ 5475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.475012 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.477454 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.477461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.477717 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.477779 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 15:47:51.477932 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:47:51.478166 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57903,"thread_start_us":65579,"threads_started":1}
I20260929 15:47:51.479550 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.407178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.0.97.2:37821)
0929 15:47:51.461814 (+ 54636us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.461828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.465068 (+ 3240us) server_negotiation.cc:407] Connection header received
0929 15:47:51.465244 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.465257 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.465607 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.465848 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:47:51.467223 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.469273 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.475230 (+ 5957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.476883 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.477021 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.477289 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.479004 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.479217 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54382,"thread_start_us":56742,"threads_started":1}
I20260929 15:47:51.484571 13598 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 (DEFAULT_TABLE table=test-table [id=a113a0a97bb64f4984ff3434b3994136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:47:51.486208 13598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:51.489885 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.406918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.0.97.1:33065)
0929 15:47:51.473337 (+ 66419us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.473347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.473380 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:47:51.473488 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.473497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.473796 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.475507 (+ 1711us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:47:51.475853 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.477603 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.480215 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.485815 (+ 5600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.485965 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.486211 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.489341 (+ 3130us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.489540 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66211,"thread_start_us":66032,"threads_started":1}
I20260929 15:47:51.489921 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.406189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60784)
0929 15:47:51.462552 (+ 56363us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.462615 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:51.462871 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:51.474692 (+ 11821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:51.474697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:51.474725 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:47:51.475185 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.475208 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.477785 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.477790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.479937 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.479956 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.486541 (+ 6585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.486549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.486737 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.486774 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:47:51.489280 (+ 2506us) client_negotiation.cc:240] Negotiation successful
0929 15:47:51.489458 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55991,"mutex_wait_us":1,"thread_start_us":66395,"threads_started":1}
I20260929 15:47:51.490307 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.423027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.0.97.3:32895)
0929 15:47:51.441618 (+ 18591us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.441626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.474300 (+ 32674us) server_negotiation.cc:407] Connection header received
0929 15:47:51.474468 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.474475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.474703 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.474908 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:47:51.483108 (+ 8200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.484780 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.487269 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.488772 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.488904 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.489119 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.489747 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.489921 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18419,"thread_start_us":18629,"threads_started":1}
I20260929 15:47:51.491070 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.422719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32895 (local address 127.0.0.1:34430)
0929 15:47:51.474157 (+ 51438us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.474177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:51.474321 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:51.475206 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:51.475210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:51.475240 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:47:51.475575 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.475586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.485013 (+ 9427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.485019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.487048 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.487064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.489303 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.489310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.489505 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.489539 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:47:51.490588 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0929 15:47:51.490741 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51306,"thread_start_us":40394,"threads_started":1}
I20260929 15:47:51.495016 13453 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 (DEFAULT_TABLE table=test-table [id=a113a0a97bb64f4984ff3434b3994136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:47:51.495213 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 (DEFAULT_TABLE table=test-table [id=a113a0a97bb64f4984ff3434b3994136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:47:51.496908 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:51.496937 13453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:51.519295 13824 tablet_bootstrap.cc:492] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Bootstrap starting.
I20260929 15:47:51.521694 13827 tablet_bootstrap.cc:492] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Bootstrap starting.
I20260929 15:47:51.522181 13826 tablet_bootstrap.cc:492] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Bootstrap starting.
I20260929 15:47:51.524139 13824 tablet_bootstrap.cc:654] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:51.525679 13824 log.cc:824] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:51.526170 13827 tablet_bootstrap.cc:654] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:51.527091 13826 tablet_bootstrap.cc:654] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:51.527526 13827 log.cc:824] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:51.528775 13826 log.cc:824] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:51.536599 13824 tablet_bootstrap.cc:492] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: No bootstrap required, opened a new log
I20260929 15:47:51.537029 13827 tablet_bootstrap.cc:492] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: No bootstrap required, opened a new log
I20260929 15:47:51.537071 13824 ts_tablet_manager.cc:1397] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.010s
I20260929 15:47:51.537415 13827 ts_tablet_manager.cc:1397] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260929 15:47:51.538414 13826 tablet_bootstrap.cc:492] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: No bootstrap required, opened a new log
I20260929 15:47:51.538762 13826 ts_tablet_manager.cc:1397] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260929 15:47:51.548683 13824 raft_consensus.cc:348] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.549418 13824 raft_consensus.cc:374] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:51.549592 13824 raft_consensus.cc:729] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f2ba4a11344aeea65655944e60eaa0, State: Initialized, Role: FOLLOWER
I20260929 15:47:51.549863 13826 raft_consensus.cc:348] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.550370 13826 raft_consensus.cc:374] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:51.550266 13827 raft_consensus.cc:348] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.550508 13826 raft_consensus.cc:729] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4422b6a0e04c09a65782c4d616a4ef, State: Initialized, Role: FOLLOWER
I20260929 15:47:51.550509 13824 consensus_queue.cc:260] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.550737 13827 raft_consensus.cc:374] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:51.550822 13827 raft_consensus.cc:729] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93767679c0f4da3b522eaa4dd12f4cc, State: Initialized, Role: FOLLOWER
I20260929 15:47:51.551168 13826 consensus_queue.cc:260] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.551481 13827 consensus_queue.cc:260] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.559266 13826 ts_tablet_manager.cc:1428] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260929 15:47:51.559266 13824 ts_tablet_manager.cc:1428] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Time spent starting tablet: real 0.022s user 0.017s sys 0.002s
I20260929 15:47:51.559563 13827 ts_tablet_manager.cc:1428] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Time spent starting tablet: real 0.022s user 0.005s sys 0.013s
I20260929 15:47:51.559707 13518 heartbeater.cc:503] Master 127.0.97.62:43063 was elected leader, sending a full tablet report...
I20260929 15:47:51.559978 13797 heartbeater.cc:503] Master 127.0.97.62:43063 was elected leader, sending a full tablet report...
I20260929 15:47:51.639973 13833 raft_consensus.cc:482] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:47:51.640373 13833 raft_consensus.cc:504] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.642426 13833 leader_election.cc:288] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e4422b6a0e04c09a65782c4d616a4ef (127.0.97.1:33065), e7f2ba4a11344aeea65655944e60eaa0 (127.0.97.2:37821)
I20260929 15:47:51.649243 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.643507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57871 (local address 127.0.97.1:33065)
0929 15:47:51.643708 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.643713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.643733 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:47:51.643813 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.643818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.643888 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.643977 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:47:51.644456 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.645695 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.647452 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.648322 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.648468 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.648575 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.648855 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.648986 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:47:51.649245 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.642669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.97.3:57871)
0929 15:47:51.643068 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.643089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:51.643212 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:51.644008 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:51.644016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:51.644043 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:47:51.644285 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.644301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.645855 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.645861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.647282 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.647299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.648499 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.648504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.648685 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.648713 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 15:47:51.648817 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:47:51.648956 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 15:47:51.650787 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.642966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39427 (local address 127.0.97.2:37821)
0929 15:47:51.643214 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.643223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.643912 (+ 689us) server_negotiation.cc:407] Connection header received
0929 15:47:51.644143 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.644148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.644267 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.644336 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:47:51.645064 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.646000 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.648550 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.649790 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.649931 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.650030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.650399 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.650537 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:47:51.650591 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b3d9313cb4a878bcb66f7f9c1fce4" candidate_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" is_pre_election: true
I20260929 15:47:51.650876 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.642711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37821 (local address 127.0.97.3:39427)
0929 15:47:51.643791 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.643807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:51.643977 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:51.644526 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:51.644529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:51.644550 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:47:51.644903 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.644911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.646200 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.646205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.648356 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:51.648371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.649997 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.650002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:51.650189 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.650235 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 15:47:51.650340 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:47:51.650484 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":2,"thread_start_us":693,"threads_started":1}
I20260929 15:47:51.651252 13473 raft_consensus.cc:2454] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93767679c0f4da3b522eaa4dd12f4cc in term 0.
I20260929 15:47:51.651993 13686 leader_election.cc:302] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4422b6a0e04c09a65782c4d616a4ef, f93767679c0f4da3b522eaa4dd12f4cc; no voters:
I20260929 15:47:51.652232 13618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b3d9313cb4a878bcb66f7f9c1fce4" candidate_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f2ba4a11344aeea65655944e60eaa0" is_pre_election: true
I20260929 15:47:51.652449 13833 raft_consensus.cc:2781] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:47:51.652606 13833 raft_consensus.cc:482] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:47:51.652707 13833 raft_consensus.cc:3037] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:51.652829 13618 raft_consensus.cc:2454] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93767679c0f4da3b522eaa4dd12f4cc in term 0.
I20260929 15:47:51.655145 13833 raft_consensus.cc:504] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.655965 13833 leader_election.cc:288] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [CANDIDATE]: Term 1 election: Requested vote from peers 5e4422b6a0e04c09a65782c4d616a4ef (127.0.97.1:33065), e7f2ba4a11344aeea65655944e60eaa0 (127.0.97.2:37821)
I20260929 15:47:51.656535 13473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b3d9313cb4a878bcb66f7f9c1fce4" candidate_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4422b6a0e04c09a65782c4d616a4ef"
I20260929 15:47:51.656766 13473 raft_consensus.cc:3037] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:51.656971 13618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b3d9313cb4a878bcb66f7f9c1fce4" candidate_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f2ba4a11344aeea65655944e60eaa0"
I20260929 15:47:51.657275 13618 raft_consensus.cc:3037] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:51.659097 13473 raft_consensus.cc:2454] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93767679c0f4da3b522eaa4dd12f4cc in term 1.
I20260929 15:47:51.659642 13686 leader_election.cc:302] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4422b6a0e04c09a65782c4d616a4ef, f93767679c0f4da3b522eaa4dd12f4cc; no voters:
I20260929 15:47:51.659981 13833 raft_consensus.cc:2781] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:47:51.660425 13618 raft_consensus.cc:2454] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93767679c0f4da3b522eaa4dd12f4cc in term 1.
I20260929 15:47:51.660607 13833 raft_consensus.cc:686] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 1 LEADER]: Becoming Leader. State: Replica: f93767679c0f4da3b522eaa4dd12f4cc, State: Running, Role: LEADER
I20260929 15:47:51.661213 13833 consensus_queue.cc:237] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } }
I20260929 15:47:51.666328 13318 catalog_manager.cc:5495] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc reported cstate change: term changed from 0 to 1, leader changed from <none> to f93767679c0f4da3b522eaa4dd12f4cc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93767679c0f4da3b522eaa4dd12f4cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32895 } health_report { overall_health: HEALTHY } } }
W20260929 15:47:51.680536 13664 tablet.cc:2369] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:47:51.727425 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.721940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.0.97.3:32895)
0929 15:47:51.722127 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.722134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.722210 (+ 76us) server_negotiation.cc:407] Connection header received
0929 15:47:51.722465 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.722469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.722546 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.722700 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:47:51.723082 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.723810 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.725240 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.725638 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.725781 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.726301 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:47:51.726830 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:47:51.726987 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.727064 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.727149 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:47:51.737807 13473 raft_consensus.cc:1261] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 1 FOLLOWER]: Refusing update from remote peer f93767679c0f4da3b522eaa4dd12f4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:47:51.737811 13618 raft_consensus.cc:1261] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 1 FOLLOWER]: Refusing update from remote peer f93767679c0f4da3b522eaa4dd12f4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:47:51.738888 13833 consensus_queue.cc:1035] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4422b6a0e04c09a65782c4d616a4ef" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:47:51.739210 13835 consensus_queue.cc:1035] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f2ba4a11344aeea65655944e60eaa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:47:51.743593 13519 tablet.cc:2369] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:47:51.792778 13318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:47:51.793133 13318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:47:51.797854 13318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043492bbb8604477852ad37a4ef15d2c: Sending DeleteTablet for 3 replicas of tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4
I20260929 15:47:51.799219 13598 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:47:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41986
I20260929 15:47:51.799219 13453 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:47:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260929 15:47:51.799504 13732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:47:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34430
I20260929 15:47:51.806761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13358
I20260929 15:47:51.814587 13850 tablet_replica.cc:330] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: stopping tablet replica
I20260929 15:47:51.816111 13850 raft_consensus.cc:2229] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:47:51.816982 13850 raft_consensus.cc:2258] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:47:51.819504 13850 ts_tablet_manager.cc:1905] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:47:51.823856 13848 tablet_replica.cc:330] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: stopping tablet replica
I20260929 15:47:51.825006 13849 tablet_replica.cc:330] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: stopping tablet replica
I20260929 15:47:51.825256 13848 raft_consensus.cc:2229] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:47:51.825443 13849 raft_consensus.cc:2229] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:47:51.825968 13848 raft_consensus.cc:2258] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:47:51.826218 13849 raft_consensus.cc:2258] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:47:51.830210 13848 ts_tablet_manager.cc:1905] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P 5e4422b6a0e04c09a65782c4d616a4ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:47:51.831422 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.808706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.0.97.1:33065)
0929 15:47:51.809010 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.809020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.809049 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:47:51.809153 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.809162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.809308 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.809469 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:51.812735 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.814212 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.818212 (+ 4000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.818898 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.819070 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.823205 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:51.823229 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:51.823256 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:51.823311 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:51.827688 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.828744 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.828754 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.828758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.828935 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.829423 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.829470 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.829474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.829959 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:51.830306 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.830745 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.830989 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 15:47:51.831496 13849 ts_tablet_manager.cc:1905] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:47:51.832393 13513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:51.835192 13850 ts_tablet_manager.cc:1918] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:47:51.835421 13850 log.cc:1196] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/0d3b3d9313cb4a878bcb66f7f9c1fce4
I20260929 15:47:51.836158 13850 ts_tablet_manager.cc:1939] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P f93767679c0f4da3b522eaa4dd12f4cc: Deleting consensus metadata
I20260929 15:47:51.838251 13305 catalog_manager.cc:4841] TS f93767679c0f4da3b522eaa4dd12f4cc (127.0.97.3:32895): tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 (table test-table [id=a113a0a97bb64f4984ff3434b3994136]) successfully deleted
I20260929 15:47:51.838634 13849 ts_tablet_manager.cc:1918] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:47:51.838857 13849 log.cc:1196] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/0d3b3d9313cb4a878bcb66f7f9c1fce4
I20260929 15:47:51.839442 13849 ts_tablet_manager.cc:1939] T 0d3b3d9313cb4a878bcb66f7f9c1fce4 P e7f2ba4a11344aeea65655944e60eaa0: Deleting consensus metadata
I20260929 15:47:51.842250 13306 catalog_manager.cc:4841] TS e7f2ba4a11344aeea65655944e60eaa0 (127.0.97.2:37821): tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4 (table test-table [id=a113a0a97bb64f4984ff3434b3994136]) successfully deleted
I20260929 15:47:51.933176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13358
W20260929 15:47:51.958210 13307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33065 (error 108)
I20260929 15:47:51.959059 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13522
I20260929 15:47:51.960309 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.959690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60794)
0929 15:47:51.959986 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:51.960102 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260929 15:47:51.960628 13307 catalog_manager.cc:4584] TS 5e4422b6a0e04c09a65782c4d616a4ef (127.0.97.1:33065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d3b3d9313cb4a878bcb66f7f9c1fce4: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
I20260929 15:47:51.975682 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:51.959920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.97.2:37821)
0929 15:47:51.960097 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:51.960103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:51.960219 (+ 116us) server_negotiation.cc:407] Connection header received
0929 15:47:51.960441 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:51.960446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:51.960535 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:51.960653 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:51.962315 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.963038 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.965071 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:51.965492 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:51.965606 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:51.969214 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:51.969231 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:51.969258 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:51.969298 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:51.972536 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.973519 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.973527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.973530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.973682 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:51.974141 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:51.974145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:51.974147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:51.974521 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:51.974793 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:51.975136 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 15:47:51.975313 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:47:51.976476 13658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:52.014876 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.014173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60800)
0929 15:47:52.014439 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.014564 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 15:47:52.016230 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.015758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60814)
0929 15:47:52.015938 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.016029 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 15:47:52.069622 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13522
I20260929 15:47:52.077114 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.076499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60822)
0929 15:47:52.076762 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.076883 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 15:47:52.078483 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.077976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60836)
0929 15:47:52.078205 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.078322 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 15:47:52.086989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13667
I20260929 15:47:52.102864 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.087806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.0.97.3:32895)
0929 15:47:52.088029 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:52.088035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:52.088055 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:47:52.088236 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:52.088243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:52.088323 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:52.088449 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:52.089860 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:52.090806 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:52.092285 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:52.092806 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:52.092960 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:52.095638 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:52.095652 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:52.095667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:52.095707 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:52.099265 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:52.100419 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:52.100426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:52.100429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:52.100597 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:52.101083 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:52.101089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:52.101092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:52.101579 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:52.101892 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:52.102246 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 15:47:52.102482 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 15:47:52.103608 13792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:52.159655 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.159075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60852)
0929 15:47:52.159329 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.159438 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 15:47:52.160811 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.160436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60868)
0929 15:47:52.160641 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.160708 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 15:47:52.227538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13667
I20260929 15:47:52.247475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13288
I20260929 15:47:52.248852 13351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:52.365752 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:52.365162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33065 (local address 127.0.0.1:60880)
0929 15:47:52.365395 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:52.365490 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 15:47:52.365721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13288
I20260929 15:47:52.387841 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 15:47:52.725060 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 15:47:52.725271 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 15:47:53.071868 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 15:47:53.072 UTC [12339] LOG: received smart shutdown request
2026-09-29 15:47:53.075 UTC [12339] LOG: background worker "logical replication launcher" (PID 12348) exited with exit code 1
2026-09-29 15:47:53.076 UTC [12343] LOG: shutting down
2026-09-29 15:47:53.258 UTC [12339] LOG: database system is shut down
2026-09-29T15:47:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56669 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:47:53 dist-test-slave-7491 krb5kdc[13864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:47:53 dist-test-slave-7491 krb5kdc[13864](info): set up 2 sockets
Sep 29 15:47:53 dist-test-slave-7491 krb5kdc[13864](info): commencing operation
krb5kdc: starting...
W20260929 15:47:54.516782 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:47:54 dist-test-slave-7491 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:47:54.768875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39897
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39897,127.0.97.61:46143,127.0.97.60:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:47:55.117996 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:55.118337 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:55.118435 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:47:55.126289 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:47:55.126435 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:55.126492 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:55.126547 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:47:55.126595 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:47:55.138489 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39897,127.0.97.61:46143,127.0.97.60:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:55.139921 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:55.142151 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:55.155467 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:55.155471 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:55.155637 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:55.156358 13877 server_base.cc:1034] running on GCE node
I20260929 15:47:55.157230 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:55.158906 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:55.160238 13877 hybrid_clock.cc:648] HybridClock initialized: now 1790696875160103 us; error 84 us; skew 500 ppm
I20260929 15:47:55.160652 13877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:55 dist-test-slave-7491 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:55.169715 13877 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:47:55.171700 13877 webserver.cc:460] Webserver started at http://127.0.97.62:43823/ using document root <none> and password file <none>
I20260929 15:47:55.172585 13877 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:55.172708 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:55.172978 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:47:55.176136 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "1ecb9ad75c0440119fbf66ba77493e6d"
format_stamp: "Formatted at 2026-09-29 15:47:55 on dist-test-slave-7491"
server_key: "08f56e56a05f74f125d0b505387912e8"
server_key_iv: "46419aa1ff90afad7b59b1cd56f1aa1a"
server_key_version: "encryptionkey@0"
I20260929 15:47:55.177029 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "1ecb9ad75c0440119fbf66ba77493e6d"
format_stamp: "Formatted at 2026-09-29 15:47:55 on dist-test-slave-7491"
server_key: "08f56e56a05f74f125d0b505387912e8"
server_key_iv: "46419aa1ff90afad7b59b1cd56f1aa1a"
server_key_version: "encryptionkey@0"
I20260929 15:47:55.182910 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:47:55.189060 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:55.190850 13877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:47:55.191042 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "1ecb9ad75c0440119fbf66ba77493e6d"
format_stamp: "Formatted at 2026-09-29 15:47:55 on dist-test-slave-7491"
server_key: "08f56e56a05f74f125d0b505387912e8"
server_key_iv: "46419aa1ff90afad7b59b1cd56f1aa1a"
server_key_version: "encryptionkey@0"
I20260929 15:47:55.191258 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:55.206732 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:55.211992 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:55.212354 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:55.232971 13877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39897
I20260929 15:47:55.232993 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39897 every 8 connection(s)
I20260929 15:47:55.234531 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:47:55.238633 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:55.243935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13877
I20260929 15:47:55.244230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:47:55.245020 388 external_mini_cluster.cc:1428] Setting key 22df447c8a755edb0ffa9f2f125338c2
I20260929 15:47:55.247145 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:47:55 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 15:47:55.277088 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.253278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.0.97.62:39897)
0929 15:47:55.254084 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:55.254097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:55.254159 (+ 62us) server_negotiation.cc:407] Connection header received
0929 15:47:55.256584 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:55.256592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:55.256866 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:55.257058 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:55.258977 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.260369 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.263047 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.263707 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.263881 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.269057 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:55.269067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:55.269082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:55.269122 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:55.272255 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.273267 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.273275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.273278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.273447 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.273981 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.273985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.273988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.274589 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:55.274785 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:55.275506 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 15:47:55.275842 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260929 15:47:55.277118 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.251664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.0.1:37504)
0929 15:47:55.252732 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.252792 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:55.256017 (+ 3225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:55.257166 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:55.257176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:55.258127 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:55.258717 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.258748 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.260558 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.260567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.262817 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.262847 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.264019 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.264029 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.264308 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.265282 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:55.265333 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:55.268781 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:55.272423 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.272435 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.272452 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.273094 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.273579 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.273587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.273593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.273827 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.274761 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:55.274772 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:55.275051 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 15:47:55.275373 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 15:47:55.276081 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":122,"thread_start_us":520,"threads_started":1}
I20260929 15:47:55.280736 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:47:55.282073 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.281649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47692)
0929 15:47:55.281833 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.281948 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 15:47:55.282397 13896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
I20260929 15:47:55.284293 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.283811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47708)
0929 15:47:55.283987 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.284085 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 15:47:55.284502 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46143: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 15:47:55.301344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46143
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39897,127.0.97.61:46143,127.0.97.60:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:47:55.333895 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } attempt: 1
I20260929 15:47:55.335055 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.334610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47716)
0929 15:47:55.334809 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.334905 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 15:47:55.336058 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.335711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47722)
0929 15:47:55.335853 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.335928 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:47:55.336289 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46143: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
I20260929 15:47:55.404528 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } attempt: 2
I20260929 15:47:55.405702 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.405228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47732)
0929 15:47:55.405478 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.405564 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 15:47:55.406644 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.406298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47734)
0929 15:47:55.406435 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.406517 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 15:47:55.406818 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46143: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
I20260929 15:47:55.498075 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } attempt: 3
I20260929 15:47:55.499251 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.498791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47746)
0929 15:47:55.499002 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.499096 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 15:47:55.500296 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.499878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47752)
0929 15:47:55.500020 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.500093 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 15:47:55.500478 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46143: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
I20260929 15:47:55.643735 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } attempt: 4
W20260929 15:47:55.643714 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:55.644057 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:55.644243 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:47:55.644986 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.644463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47754)
0929 15:47:55.644772 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.644857 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 15:47:55.645947 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.645602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47758)
0929 15:47:55.645742 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.645834 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 15:47:55.646142 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46143: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
W20260929 15:47:55.652055 13951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:47:55.652279 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:55.652384 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:55.652488 13951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:47:55.652561 13951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:47:55.663491 13951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39897,127.0.97.61:46143,127.0.97.60:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:55.665002 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:55.666358 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:55.679018 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:55.679041 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:55.678967 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:55.680280 13951 server_base.cc:1034] running on GCE node
I20260929 15:47:55.681290 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:55.682921 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:55.684253 13951 hybrid_clock.cc:648] HybridClock initialized: now 1790696875684155 us; error 44 us; skew 500 ppm
I20260929 15:47:55.684659 13951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:55 dist-test-slave-7491 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:55.688341 13951 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 15:47:55.690070 13951 webserver.cc:460] Webserver started at http://127.0.97.61:36745/ using document root <none> and password file <none>
I20260929 15:47:55.691054 13951 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:55.691246 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:55.691586 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:47:55.695217 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data/instance:
uuid: "8ecdfdae3b954f9aad86af263b93db4a"
format_stamp: "Formatted at 2026-09-29 15:47:55 on dist-test-slave-7491"
server_key: "84e842087a79c29803d542e109369567"
server_key_iv: "b50cc3704acfee008af1902223172d59"
server_key_version: "encryptionkey@0"
I20260929 15:47:55.696132 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal/instance:
uuid: "8ecdfdae3b954f9aad86af263b93db4a"
format_stamp: "Formatted at 2026-09-29 15:47:55 on dist-test-slave-7491"
server_key: "84e842087a79c29803d542e109369567"
server_key_iv: "b50cc3704acfee008af1902223172d59"
server_key_version: "encryptionkey@0"
I20260929 15:47:55.701898 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:47:55.705873 13966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:55.707377 13951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:47:55.707580 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal
uuid: "8ecdfdae3b954f9aad86af263b93db4a"
format_stamp: "Formatted at 2026-09-29 15:47:55 on dist-test-slave-7491"
server_key: "84e842087a79c29803d542e109369567"
server_key_iv: "b50cc3704acfee008af1902223172d59"
server_key_version: "encryptionkey@0"
I20260929 15:47:55.707829 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:55.738158 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:55.743573 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:55.743920 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:55.765412 13951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46143
I20260929 15:47:55.765412 14017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46143 every 8 connection(s)
I20260929 15:47:55.766920 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:47:55.771130 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:55.775992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 13951
I20260929 15:47:55.776252 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-1/wal/instance
I20260929 15:47:55.776790 388 external_mini_cluster.cc:1428] Setting key aec268225053e8b229ff68cb231cbf4d
I20260929 15:47:55.779992 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:47:55 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 15:47:55.804849 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.783906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.0.97.62:39897)
0929 15:47:55.784486 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:55.784494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:55.785510 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 15:47:55.787473 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:55.787479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:55.787573 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:55.787695 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:55.789335 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.790060 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.791706 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.792160 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.792340 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.797119 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:55.797130 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:55.797138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:55.797191 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:55.800899 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.801902 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.801913 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.801916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.802072 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.802797 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.802802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.802805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.803216 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:55.803383 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:55.804159 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 15:47:55.804449 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":275,"threads_started":1}
I20260929 15:47:55.805887 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.784500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.0.1:37514)
0929 15:47:55.785325 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.785378 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:55.787189 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:55.787959 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:55.787966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:55.788584 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:55.789151 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.789169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.790229 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.790235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.791485 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.791498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.792460 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.792467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.792957 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.793813 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:55.793842 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:55.796780 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:55.801188 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.801200 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.801222 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.801736 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.802267 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.802273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.802279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.802656 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.803354 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:55.803363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:55.803770 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 15:47:55.804049 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 15:47:55.804664 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":365,"threads_started":1}
I20260929 15:47:55.807694 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:47:55 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:47:55.825350 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.808732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47760)
0929 15:47:55.808944 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.808961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:55.809062 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:55.810172 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:55.810183 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:55.810720 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:55.810938 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.810953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.812510 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.812518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.814195 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.814225 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.815162 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.815174 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.815462 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.816126 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:55.816154 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:55.818823 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:55.822589 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.822597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.822604 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.822990 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.823396 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.823401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.823404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.823529 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.824190 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:55.824198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:55.824371 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 15:47:55.824512 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:47:55.824880 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 15:47:55.825394 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.809059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.0.97.61:46143)
0929 15:47:55.809742 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:55.809748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:55.809774 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:47:55.809818 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:55.809824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:55.810001 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:55.810124 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:55.811125 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.812353 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.814382 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.814989 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.815103 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.819056 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:55.819071 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:55.819086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:55.819116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:55.822418 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.823140 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.823154 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.823157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.823268 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.823645 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.823653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.823656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.824055 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:55.824319 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:55.824659 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 15:47:55.825006 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":544,"thread_start_us":389,"threads_started":1}
I20260929 15:47:55.828233 14018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:47:55.829704 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.829221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50448)
0929 15:47:55.829397 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.829546 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 15:47:55.830001 13968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 15:47:55.831729 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.831306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50458)
0929 15:47:55.831470 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.831565 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 15:47:55.832031 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:55.832871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36453
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39897,127.0.97.61:46143,127.0.97.60:36453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:47:55.881474 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 1
I20260929 15:47:55.882903 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.882244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50470)
0929 15:47:55.882623 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.882740 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 15:47:55.884505 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.883907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50478)
0929 15:47:55.884213 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.884323 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 15:47:55.884780 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:55.945438 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } attempt: 5
I20260929 15:47:55.953104 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 2
I20260929 15:47:55.954600 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.953983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50490)
0929 15:47:55.954238 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.954354 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
Sep 29 15:47:55 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:47:55.955960 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.955472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50498)
0929 15:47:55.955699 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.955798 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260929 15:47:55.956146 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:55.962687 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.946364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.97.61:46143)
0929 15:47:55.946574 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:55.946581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:55.946701 (+ 120us) server_negotiation.cc:407] Connection header received
0929 15:47:55.946889 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:55.946894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:55.946981 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:55.947108 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:55.948005 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.949093 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.950618 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.951528 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.951711 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.956589 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:55.956595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:55.956598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:55.956663 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:55.959339 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.960389 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.960395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.960398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.960629 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:55.961172 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:55.961177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:55.961180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:55.961565 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:55.961785 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:55.962258 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 15:47:55.962450 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:47:55.962780 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.946356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47774)
0929 15:47:55.946565 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.946599 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:55.946742 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:55.947204 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:55.947208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:55.947621 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:55.947824 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.947834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.949291 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.949301 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.950442 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:55.950459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:55.951857 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:55.951876 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:55.952113 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:55.953381 (+ 1268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:55.953448 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:55.956291 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:55.959538 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.959544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.959549 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.960157 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.960819 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:55.960823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:55.960827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:55.961028 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:55.961710 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:55.961715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:55.961902 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 15:47:55.962100 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 15:47:55.962424 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:47:55.964011 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:47:55.965341 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.964843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50514)
0929 15:47:55.965044 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.965149 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 15:47:55.966522 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:55.966124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50522)
0929 15:47:55.966283 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:55.966392 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:47:55.966792 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:56.018059 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 1
I20260929 15:47:56.019349 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.018794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50524)
0929 15:47:56.019021 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.019138 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 15:47:56.020478 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.020077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50530)
0929 15:47:56.020252 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.020341 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:47:56.020728 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:56.047442 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 3
I20260929 15:47:56.048728 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.048248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50534)
0929 15:47:56.048474 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.048555 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 15:47:56.050009 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.049487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50540)
0929 15:47:56.049785 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.049865 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260929 15:47:56.050347 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:56.088992 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 2
I20260929 15:47:56.090409 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.089782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50544)
0929 15:47:56.090062 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.090187 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 15:47:56.091678 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.091210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50556)
0929 15:47:56.091387 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.091495 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:47:56.091965 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
W20260929 15:47:56.192987 14026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:47:56.193459 14026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:47:56.193616 14026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:47:56.194857 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 4
I20260929 15:47:56.198246 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.197105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50560)
0929 15:47:56.197937 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.198029 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260929 15:47:56.198199 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 3
I20260929 15:47:56.199280 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.198805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50574)
0929 15:47:56.198987 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.199079 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 15:47:56.199545 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.199136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50590)
0929 15:47:56.199323 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.199411 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 15:47:56.199767 14018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
I20260929 15:47:56.200666 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.200234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50592)
0929 15:47:56.200449 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.200557 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 15:47:56.201126 13944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36453: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36453: connect: Connection refused (error 111)
W20260929 15:47:56.203565 14026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:47:56.203682 14026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:47:56.203730 14026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:47:56.203794 14026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:47:56.203840 14026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:47:56.214922 14026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39897,127.0.97.61:46143,127.0.97.60:36453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:47:56.216354 14026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:47:56.217684 14026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:47:56.229528 14031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:56.229633 14032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:47:56.229588 14034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:47:56.233816 14026 server_base.cc:1034] running on GCE node
I20260929 15:47:56.234745 14026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:47:56.236572 14026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:47:56.237816 14026 hybrid_clock.cc:648] HybridClock initialized: now 1790696876237756 us; error 62 us; skew 500 ppm
I20260929 15:47:56.238270 14026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:47:56.242218 14026 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 15:47:56.243817 14026 webserver.cc:460] Webserver started at http://127.0.97.60:33727/ using document root <none> and password file <none>
I20260929 15:47:56.244628 14026 fs_manager.cc:359] Metadata directory not provided
I20260929 15:47:56.244776 14026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:47:56.245110 14026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:47:56.248238 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data/instance:
uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe"
format_stamp: "Formatted at 2026-09-29 15:47:56 on dist-test-slave-7491"
server_key: "c46e782f0bd990694dc6fdb53a355ae6"
server_key_iv: "d0e3e5a3fa170621b24d1bb93d2a40b5"
server_key_version: "encryptionkey@0"
I20260929 15:47:56.249101 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal/instance:
uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe"
format_stamp: "Formatted at 2026-09-29 15:47:56 on dist-test-slave-7491"
server_key: "c46e782f0bd990694dc6fdb53a355ae6"
server_key_iv: "d0e3e5a3fa170621b24d1bb93d2a40b5"
server_key_version: "encryptionkey@0"
I20260929 15:47:56.254667 14026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:47:56.258561 14041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:47:56.260013 14026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:47:56.260284 14026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal
uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe"
format_stamp: "Formatted at 2026-09-29 15:47:56 on dist-test-slave-7491"
server_key: "c46e782f0bd990694dc6fdb53a355ae6"
server_key_iv: "d0e3e5a3fa170621b24d1bb93d2a40b5"
server_key_version: "encryptionkey@0"
I20260929 15:47:56.260511 14026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:47:56.309187 14026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:47:56.315609 14026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:47:56.315899 14026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:47:56.336805 14026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36453
I20260929 15:47:56.336827 14092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36453 every 8 connection(s)
I20260929 15:47:56.338323 14026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:47:56.340394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14026
I20260929 15:47:56.340806 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790696645767654-388-0/minicluster-data/master-2/wal/instance
I20260929 15:47:56.341689 388 external_mini_cluster.cc:1428] Setting key ee44520521f3ba4367ecd79f101f70cc
I20260929 15:47:56.344708 14093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:47:56.354203 14093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:47:56.369009 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.344027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.0.97.60:36453)
0929 15:47:56.344939 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.344955 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.345011 (+ 56us) server_negotiation.cc:407] Connection header received
0929 15:47:56.347417 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.347451 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.348036 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.348584 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.350319 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.352469 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.354019 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.355138 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.355355 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.360456 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.360463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.360475 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.360512 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.364379 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.365204 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.365214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.365230 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.365468 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.365907 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.365914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.365919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.366562 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.366829 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.367403 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.367926 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":346,"threads_started":1}
I20260929 15:47:56.374527 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.357425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.97.62:39897)
0929 15:47:56.357932 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.357937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.358530 (+ 593us) server_negotiation.cc:407] Connection header received
0929 15:47:56.358765 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.358773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.358853 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.358974 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.360295 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.361123 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.362898 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.363525 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.363680 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.367883 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.367889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.367892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.367929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.371296 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.372251 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.372259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.372263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.372456 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.372933 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.372937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.372943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.373260 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.373421 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.373895 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.374118 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":256,"threads_started":1}
I20260929 15:47:56.374550 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.357500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.0.1:37530)
0929 15:47:56.358396 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.358437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:56.358620 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:56.359084 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:56.359089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:56.359798 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:56.360094 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.360104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.361322 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.361329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.362692 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.362707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.363914 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.363922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.364083 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.364812 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:56.364830 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:56.367617 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:56.371459 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.371471 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.371478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.372079 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.372604 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.372612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.372617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.372801 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.373393 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:56.373401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:56.373598 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 15:47:56.373772 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 15:47:56.374102 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":550,"threads_started":1}
I20260929 15:47:56.376277 14093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:47:56.378358 13944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 4
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:47:56.393565 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.372445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.97.62:39897)
0929 15:47:56.373254 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.373258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.373279 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:47:56.373318 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.373321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.373375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.373500 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.374900 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.375765 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.377403 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.378226 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.378401 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.384972 (+ 6571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.384979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.384989 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.385038 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.388860 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.390290 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.390298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.390304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.390457 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.391033 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.391038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.391043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.391457 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.391790 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.392833 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.393105 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":431,"threads_started":1}
I20260929 15:47:56.394373 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.377135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.0.97.61:46143)
0929 15:47:56.377305 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.377309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.377501 (+ 192us) server_negotiation.cc:407] Connection header received
0929 15:47:56.377747 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.377754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.377852 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.377973 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.379433 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.380096 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.382241 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.382922 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.383045 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.387482 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.387487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.387491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.387528 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.391086 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.391986 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.391995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.392001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.392214 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.392910 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.392918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.392922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.393252 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.393405 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.393940 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.394126 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:47:56.395221 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.378932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50608)
0929 15:47:56.379098 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.379116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:56.379311 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:56.379703 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:56.379707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:56.380117 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:56.380337 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.380345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.381605 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.381612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.382817 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.382836 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.383912 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.383921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.384131 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.384769 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:56.384797 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:56.388001 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:56.392084 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.392089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.392093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.392550 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.392957 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.392962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.392965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.393121 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.393806 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:56.394160 (+ 354us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:56.394356 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 15:47:56.394511 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 15:47:56.394869 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 15:47:56.395394 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.377139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47786)
0929 15:47:56.377416 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.377435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:56.377533 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:56.378023 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:56.378027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:56.378509 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:56.379280 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.379290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.380266 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.380273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.382044 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.382064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.383239 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.383243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.383435 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.384536 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:56.384562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:56.387251 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:56.391240 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.391246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.391250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.391850 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.392362 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.392368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.392373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.392545 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.393388 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:56.393393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:56.393606 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 15:47:56.394634 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0929 15:47:56.394996 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":7}
I20260929 15:47:56.396377 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.378986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.0.97.60:36453)
0929 15:47:56.379178 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.379184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.379227 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:47:56.379472 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.379478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.379554 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.379679 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.380485 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.381456 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.382990 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.383745 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.383874 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.388277 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.388284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.388290 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.388333 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.391922 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.392666 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.392673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.392676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.392841 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.393252 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.393258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.393260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.393623 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.395589 (+ 1966us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.395751 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.396007 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":67200}
I20260929 15:47:56.396811 14093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:47:56.403323 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: Bootstrap starting.
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:47:56.407135 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:56.408542 13944 log.cc:824] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:56.412262 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: No bootstrap required, opened a new log
I20260929 15:47:56.412676 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.397964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.0.97.60:36453)
0929 15:47:56.398094 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.398099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.398114 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:47:56.398201 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.398205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.398274 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.398373 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.399196 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.400024 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.401292 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.402107 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.402251 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.406044 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.406059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.406063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.406097 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.409409 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.410223 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.410228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.410231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.410376 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.410794 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.410798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.410801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.411257 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.411416 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.411889 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.412267 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260929 15:47:56.412847 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.397666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50624)
0929 15:47:56.397997 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.398015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:56.398107 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:56.398386 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:56.398390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:56.398914 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:56.399072 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.399081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.400234 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.400239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.401140 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.401155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.402297 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.402303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.402493 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.403015 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:56.403037 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:56.405840 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:56.409538 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.409556 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.409560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.410075 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.410496 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.410501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.410503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.410667 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.411463 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:56.411467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:56.411599 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 15:47:56.411808 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 15:47:56.412425 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":300}
I20260929 15:47:56.414820 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.395849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.97.61:46143)
0929 15:47:56.396022 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.396029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.396051 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:47:56.396317 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.396323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.396411 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.396505 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.398149 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.399249 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.401104 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.401637 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.401783 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.407474 (+ 5691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.407483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.407489 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.407542 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.411288 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.412433 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.412438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.412442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.412622 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.413118 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.413123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.413126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.413517 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.413847 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.414181 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.414439 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:47:56.418352 14093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: Bootstrap starting.
I20260929 15:47:56.421849 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.422308 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:56.422492 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecb9ad75c0440119fbf66ba77493e6d, State: Initialized, Role: FOLLOWER
I20260929 15:47:56.422753 14093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:56.423277 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.425174 14093 log.cc:824] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:56.424952 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.425377 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.425787 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:47:56.426188 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:47:56.428597 14093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: No bootstrap required, opened a new log
I20260929 15:47:56.440979 14093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.441713 14093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:56.441906 14093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e143ab8cfb4c8ca218f9e91fb03ebe, State: Initialized, Role: FOLLOWER
I20260929 15:47:56.443686 14093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.449400 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.449913 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.452003 14093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:47:56.465363 13898 tablet.cc:2369] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:47:56.469077 14124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:47:56.469288 14124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:47:56.477309 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.434101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.97.61:46143)
0929 15:47:56.434258 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.434264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.443643 (+ 9379us) server_negotiation.cc:407] Connection header received
0929 15:47:56.444300 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.444309 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.444422 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.444515 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.447552 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.448985 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.453550 (+ 4565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.454404 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.454529 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.461978 (+ 7449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.461990 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.461997 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.462059 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.471920 (+ 9861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.474165 (+ 2245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.474170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.474174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.474378 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.474838 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.474844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.474847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.475260 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.475533 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.476632 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.476902 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 15:47:56.484977 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.439256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.0.97.60:36453)
0929 15:47:56.439488 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.439495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.443679 (+ 4184us) server_negotiation.cc:407] Connection header received
0929 15:47:56.444016 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.444024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.444128 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.444231 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.446914 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.448090 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.455834 (+ 7744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.456809 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.456956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.464763 (+ 7807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.464773 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.464779 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.464835 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.480685 (+ 15850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.481770 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.481790 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.481797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.481975 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.482490 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.482494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.482498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.482892 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.484201 (+ 1309us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.484338 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.484631 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 15:47:56.475183 13877 master_runner.cc:186] Error getting master registration for 127.0.97.61:46143: OK, instance_id { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" instance_seqno: 1790696875759269 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:47:56.488574 13877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:47:56.493207 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.432846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.0.97.62:39897)
0929 15:47:56.433196 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.433207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.443146 (+ 9939us) server_negotiation.cc:407] Connection header received
0929 15:47:56.444336 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.444343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.444432 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.444501 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.447079 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.448191 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.450779 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.451481 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.451591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.461562 (+ 9971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.461575 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.461583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.461653 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.488061 (+ 26408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.489267 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.489273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.489276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.489458 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.489961 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.489967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.489972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.490368 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.492264 (+ 1896us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.492432 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.492742 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:47:56.498998 14018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } attempt: 5
I20260929 15:47:56.487160 14026 master_runner.cc:186] Error getting master registration for 127.0.97.61:46143: OK, instance_id { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" instance_seqno: 1790696875759269 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:47:56.499548 14026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:47:56.499619 14133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:47:56.499782 14133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 15:47:56 dist-test-slave-7491 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:47:56.515906 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.500644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.0.97.60:36453)
0929 15:47:56.500839 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:47:56.500845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:47:56.500866 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:47:56.500929 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:47:56.500936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:47:56.501021 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:47:56.501116 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:47:56.502434 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.503425 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.504739 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.505209 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.505355 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.508971 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:47:56.508978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:47:56.508982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:47:56.509026 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:47:56.512948 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.513670 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.513676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.513680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.513963 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:47:56.514366 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:47:56.514371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:47:56.514374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:47:56.514664 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:47:56.514869 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 15:47:56.515225 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 15:47:56.515477 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:47:56.518092 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:56.499783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36453 (local address 127.0.0.1:50634)
0929 15:47:56.500081 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:56.500105 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:47:56.500371 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:47:56.501322 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:47:56.501326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:47:56.501986 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:47:56.502250 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.502266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.503584 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.503592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.504576 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:47:56.504590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:47:56.505383 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:47:56.505389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:47:56.505579 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:47:56.506237 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:47:56.506256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:47:56.508727 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:47:56.513097 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.513105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.513109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.513526 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.514089 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:47:56.514094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:47:56.514103 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:47:56.514247 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:47:56.514803 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:47:56.514809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:47:56.514972 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 15:47:56.515118 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:47:56.515461 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 15:47:56.523406 14018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: Bootstrap starting.
I20260929 15:47:56.526927 14018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: Neither blocks nor log segments found. Creating new log.
I20260929 15:47:56.528414 14018 log.cc:824] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:47:56.535595 14018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: No bootstrap required, opened a new log
I20260929 15:47:56.546252 14018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.546679 14018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:47:56.546818 14018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecdfdae3b954f9aad86af263b93db4a, State: Initialized, Role: FOLLOWER
I20260929 15:47:56.547521 14018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.548619 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.549098 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.549983 14018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:47:56.568264 14047 tablet.cc:2369] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:47:56.574175 13951 master_runner.cc:263] No leader master found from master 8ecdfdae3b954f9aad86af263b93db4a
I20260929 15:47:56.574373 13951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 15:47:56.575426 14147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:47:56.575661 14147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:47:56.659291 14117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:47:56.659655 14117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.661484 14117 leader_election.cc:288] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ecb9ad75c0440119fbf66ba77493e6d (127.0.97.62:39897), 8ecdfdae3b954f9aad86af263b93db4a (127.0.97.61:46143)
I20260929 15:47:56.662523 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" is_pre_election: true
I20260929 15:47:56.663192 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e143ab8cfb4c8ca218f9e91fb03ebe in term 0.
I20260929 15:47:56.664248 13993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ecdfdae3b954f9aad86af263b93db4a" is_pre_election: true
I20260929 15:47:56.664896 13993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e143ab8cfb4c8ca218f9e91fb03ebe in term 0.
I20260929 15:47:56.665918 14042 leader_election.cc:302] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ecb9ad75c0440119fbf66ba77493e6d, 94e143ab8cfb4c8ca218f9e91fb03ebe; no voters:
I20260929 15:47:56.666435 14117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:47:56.666582 14117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:47:56.666676 14117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:56.671531 14117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.673485 13919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ecb9ad75c0440119fbf66ba77493e6d"
I20260929 15:47:56.673779 13919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:56.677644 13919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e143ab8cfb4c8ca218f9e91fb03ebe in term 1.
I20260929 15:47:56.678774 13993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ecdfdae3b954f9aad86af263b93db4a"
I20260929 15:47:56.679054 13993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:47:56.682830 13993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e143ab8cfb4c8ca218f9e91fb03ebe in term 1.
I20260929 15:47:56.683096 14117 leader_election.cc:288] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [CANDIDATE]: Term 1 election: Requested vote from peers 1ecb9ad75c0440119fbf66ba77493e6d (127.0.97.62:39897), 8ecdfdae3b954f9aad86af263b93db4a (127.0.97.61:46143)
I20260929 15:47:56.683874 14045 leader_election.cc:302] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ecdfdae3b954f9aad86af263b93db4a, 94e143ab8cfb4c8ca218f9e91fb03ebe; no voters:
I20260929 15:47:56.684365 14117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:47:56.695765 14117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [term 1 LEADER]: Becoming Leader. State: Replica: 94e143ab8cfb4c8ca218f9e91fb03ebe, State: Running, Role: LEADER
I20260929 15:47:56.696837 14117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } }
I20260929 15:47:56.706517 14149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e143ab8cfb4c8ca218f9e91fb03ebe. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.707154 14149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:56.713595 14151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:47:56.716931 14151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:47:56.738840 13993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [term 1 FOLLOWER]: Refusing update from remote peer 94e143ab8cfb4c8ca218f9e91fb03ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:47:56.740154 14149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:47:56.747004 13972 tablet.cc:2369] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:47:56.748672 13919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [term 1 FOLLOWER]: Refusing update from remote peer 94e143ab8cfb4c8ca218f9e91fb03ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:47:56.754058 14149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:47:56.775425 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e143ab8cfb4c8ca218f9e91fb03ebe. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.775729 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.779342 14152 mvcc.cc:205] Tried to move back new op lower bound from 7334694407108382720 to 7334694406957641728. Current Snapshot: MvccSnapshot[applied={T|T < 7334694407108382720}]
I20260929 15:47:56.788985 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.789343 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.791031 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.791381 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:56.792276 14151 catalog_manager.cc:1347] Generated new cluster ID: 70f4840b3f034033ada7c5ecd7478791
I20260929 15:47:56.792379 14151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:47:56.795677 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e143ab8cfb4c8ca218f9e91fb03ebe. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.796030 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.818365 14149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.824331 14149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe [sys.catalog]: This master's current role is: LEADER
I20260929 15:47:56.825687 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecb9ad75c0440119fbf66ba77493e6d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } peers { permanent_uuid: "8ecdfdae3b954f9aad86af263b93db4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46143 } } peers { permanent_uuid: "94e143ab8cfb4c8ca218f9e91fb03ebe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36453 } } }
I20260929 15:47:56.825986 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:47:56.861294 14151 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:47:56.863483 14151 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:47:56.888370 14151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe: Generated new TSK 0
I20260929 15:47:56.889700 14151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:47:57.471819 14124 catalog_manager.cc:1259] Loaded cluster ID: 70f4840b3f034033ada7c5ecd7478791
I20260929 15:47:57.471997 14124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: loading cluster ID for follower catalog manager: success
I20260929 15:47:57.474764 14124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: acquiring CA information for follower catalog manager: success
I20260929 15:47:57.476853 14124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ecb9ad75c0440119fbf66ba77493e6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:47:57.577837 14147 catalog_manager.cc:1259] Loaded cluster ID: 70f4840b3f034033ada7c5ecd7478791
I20260929 15:47:57.577945 14147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: loading cluster ID for follower catalog manager: success
I20260929 15:47:57.580518 14147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: acquiring CA information for follower catalog manager: success
I20260929 15:47:57.582453 14147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ecdfdae3b954f9aad86af263b93db4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:47:58.042080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13877
I20260929 15:47:58.043725 13939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:58.164389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13877
I20260929 15:47:58.189393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 13951
I20260929 15:47:58.190864 14013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:58.301798 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:58.300901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.0.1:37560)
0929 15:47:58.301491 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:58.301625 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":297,"threads_started":1}
W20260929 15:47:58.302193 14042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39897: connect: Connection refused (error 111)
I20260929 15:47:58.307366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 13951
I20260929 15:47:58.308507 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:58.307310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.0.1:37570)
0929 15:47:58.307484 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:58.308299 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 15:47:58.308807 14042 consensus_peers.cc:487] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe -> Peer 1ecb9ad75c0440119fbf66ba77493e6d (127.0.97.62:39897): Couldn't send request to peer 1ecb9ad75c0440119fbf66ba77493e6d. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 15:47:58.330735 14045 connection.cc:535] client connection to 127.0.97.61:46143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:47:58.331169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14026
I20260929 15:47:58.332387 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:47:58.331781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46143 (local address 127.0.0.1:47814)
0929 15:47:58.332083 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 15:47:58.332225 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260929 15:47:58.332624 14045 consensus_peers.cc:487] T 00000000000000000000000000000000 P 94e143ab8cfb4c8ca218f9e91fb03ebe -> Peer 8ecdfdae3b954f9aad86af263b93db4a (127.0.97.61:46143): Couldn't send request to peer 8ecdfdae3b954f9aad86af263b93db4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:47:58.332790 14088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:47:58.451107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14026
2026-09-29T15:47:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5064 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:47:58 dist-test-slave-7491 krb5kdc[14172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:47:58 dist-test-slave-7491 krb5kdc[14172](info): set up 2 sockets
Sep 29 15:47:58 dist-test-slave-7491 krb5kdc[14172](info): commencing operation
krb5kdc: starting...
W20260929 15:47:59.584830 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:47:59 dist-test-slave-7491 krb5kdc[14172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:47:59.832922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44381
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:00.181453 14185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:00.181803 14185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:00.181980 14185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:00.191661 14185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:48:00.191812 14185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:00.191917 14185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:00.192028 14185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:48:00.192191 14185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:48:00.202953 14185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:00.204460 14185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:00.205855 14185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:00.219679 14191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:00.219775 14193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:00.219724 14190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:00.220847 14185 server_base.cc:1034] running on GCE node
I20260929 15:48:00.221716 14185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:00.223304 14185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:00.224506 14185 hybrid_clock.cc:648] HybridClock initialized: now 1790696880224455 us; error 52 us; skew 500 ppm
I20260929 15:48:00.224884 14185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:00 dist-test-slave-7491 krb5kdc[14172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:00.240640 14185 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:48:00.242391 14185 webserver.cc:460] Webserver started at http://127.0.97.62:43641/ using document root <none> and password file <none>
I20260929 15:48:00.243130 14185 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:00.243239 14185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:00.243506 14185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:00.246820 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "02bb9b585ea046c39a87777883093260"
format_stamp: "Formatted at 2026-09-29 15:48:00 on dist-test-slave-7491"
server_key: "0efad1d7f2bfdc12d1cec3f6606c5198"
server_key_iv: "5b682d4540e0c51237b263236183da0e"
server_key_version: "encryptionkey@0"
I20260929 15:48:00.247619 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "02bb9b585ea046c39a87777883093260"
format_stamp: "Formatted at 2026-09-29 15:48:00 on dist-test-slave-7491"
server_key: "0efad1d7f2bfdc12d1cec3f6606c5198"
server_key_iv: "5b682d4540e0c51237b263236183da0e"
server_key_version: "encryptionkey@0"
I20260929 15:48:00.253314 14185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:48:00.257472 14200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:00.259017 14185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:48:00.259198 14185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "02bb9b585ea046c39a87777883093260"
format_stamp: "Formatted at 2026-09-29 15:48:00 on dist-test-slave-7491"
server_key: "0efad1d7f2bfdc12d1cec3f6606c5198"
server_key_iv: "5b682d4540e0c51237b263236183da0e"
server_key_version: "encryptionkey@0"
I20260929 15:48:00.259398 14185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:00.282743 14185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:00.287405 14185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:00.287690 14185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:00.311015 14251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44381 every 8 connection(s)
I20260929 15:48:00.311101 14185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44381
I20260929 15:48:00.312781 14185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:48:00.317518 14252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:00.318404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14185
I20260929 15:48:00.318636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:48:00.319307 388 external_mini_cluster.cc:1428] Setting key 24d0fbfdd895f638fbe4e9dc4a467bb2
I20260929 15:48:00.333571 14252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260: Bootstrap starting.
Sep 29 15:48:00 dist-test-slave-7491 krb5kdc[14172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:00.337448 14252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:00.338641 14252 log.cc:824] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:00.342657 14252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260: No bootstrap required, opened a new log
I20260929 15:48:00.349279 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:00.321526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.0.97.62:44381)
0929 15:48:00.322438 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:00.322464 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:00.322519 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:48:00.325036 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:00.325073 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:00.325695 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:00.326237 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:00.328648 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:00.330823 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:00.332518 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:00.333640 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:00.333895 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:00.338747 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:00.338761 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:00.338784 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:00.338844 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:00.343325 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:00.344735 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:00.344754 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:00.344764 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:00.345007 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:00.345482 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:00.345488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:00.345492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:00.346325 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:00.346654 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:00.347194 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 15:48:00.347693 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":402,"threads_started":1}
I20260929 15:48:00.353401 14252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bb9b585ea046c39a87777883093260" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44381 } }
I20260929 15:48:00.353840 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:48:00.353890 14252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:00.353996 14252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bb9b585ea046c39a87777883093260, State: Initialized, Role: FOLLOWER
I20260929 15:48:00.354561 14252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bb9b585ea046c39a87777883093260" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44381 } }
I20260929 15:48:00.354805 14252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:48:00.354940 14252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:48:00.355120 14252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:00.358201 14252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bb9b585ea046c39a87777883093260" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44381 } }
I20260929 15:48:00.358760 14252 leader_election.cc:302] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02bb9b585ea046c39a87777883093260; no voters:
I20260929 15:48:00.359393 14252 leader_election.cc:288] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:48:00.359647 14257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:00.360386 14257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [term 1 LEADER]: Becoming Leader. State: Replica: 02bb9b585ea046c39a87777883093260, State: Running, Role: LEADER
I20260929 15:48:00.360921 14257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bb9b585ea046c39a87777883093260" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44381 } }
I20260929 15:48:00.361780 14252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:48:00.363801 14259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02bb9b585ea046c39a87777883093260. Latest consensus state: current_term: 1 leader_uuid: "02bb9b585ea046c39a87777883093260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bb9b585ea046c39a87777883093260" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44381 } } }
I20260929 15:48:00.364579 14259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:00.365180 14258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02bb9b585ea046c39a87777883093260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02bb9b585ea046c39a87777883093260" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44381 } } }
I20260929 15:48:00.365767 14258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:00.367857 14262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:48:00.374101 14262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:48:00.384394 14262 catalog_manager.cc:1347] Generated new cluster ID: e2703003e415449792abc1d55ff301ef
I20260929 15:48:00.384534 14262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:48:00.395210 14262 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:48:00.396767 14262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:48:00.406181 14262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02bb9b585ea046c39a87777883093260: Generated new TSK 0
I20260929 15:48:00.407094 14262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:48:00.416194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14185
I20260929 15:48:00.417548 14247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:00.509487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14185
2026-09-29T15:48:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2053 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:48:00 dist-test-slave-7491 krb5kdc[14280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:48:00 dist-test-slave-7491 krb5kdc[14280](info): set up 2 sockets
Sep 29 15:48:00 dist-test-slave-7491 krb5kdc[14280](info): commencing operation
krb5kdc: starting...
W20260929 15:48:01.616681 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:48:01 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:48:01.859560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39295
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:02.204689 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:02.205029 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:02.205143 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:02.212666 14293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:48:02.212788 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:02.212836 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:02.212895 14293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:48:02.212940 14293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:48:02.224896 14293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:02.226258 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:02.227532 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:02.239673 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:02.239652 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:02.239703 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:02.240528 14293 server_base.cc:1034] running on GCE node
I20260929 15:48:02.241350 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:02.242815 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:02.244029 14293 hybrid_clock.cc:648] HybridClock initialized: now 1790696882243985 us; error 33 us; skew 500 ppm
I20260929 15:48:02.244428 14293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:02 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:02.247730 14293 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:48:02.249492 14293 webserver.cc:460] Webserver started at http://127.0.97.62:34479/ using document root <none> and password file <none>
I20260929 15:48:02.250239 14293 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:02.250401 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:02.250710 14293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:02.254005 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "e5b3b5a7abd64ff09a438d260b2bea52"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "f248af9947876c958548590989b6a55e"
server_key_iv: "cbb4ad8f20f40991f4be0d91e081d3a1"
server_key_version: "encryptionkey@0"
I20260929 15:48:02.254863 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "e5b3b5a7abd64ff09a438d260b2bea52"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "f248af9947876c958548590989b6a55e"
server_key_iv: "cbb4ad8f20f40991f4be0d91e081d3a1"
server_key_version: "encryptionkey@0"
I20260929 15:48:02.260634 14293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 15:48:02.264781 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:02.266230 14293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:48:02.266450 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "e5b3b5a7abd64ff09a438d260b2bea52"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "f248af9947876c958548590989b6a55e"
server_key_iv: "cbb4ad8f20f40991f4be0d91e081d3a1"
server_key_version: "encryptionkey@0"
I20260929 15:48:02.266669 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:02.287045 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:02.291930 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:02.292330 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:02.312131 14293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39295
I20260929 15:48:02.312153 14359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39295 every 8 connection(s)
I20260929 15:48:02.313624 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:48:02.314932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14293
I20260929 15:48:02.315160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:48:02.315964 388 external_mini_cluster.cc:1428] Setting key d86285b36dad46bfaf627323a39c8f74
I20260929 15:48:02.318640 14360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:48:02 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:02.332967 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap starting.
I20260929 15:48:02.336347 14360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:02.337461 14360 log.cc:824] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:02.340670 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: No bootstrap required, opened a new log
I20260929 15:48:02.340806 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:02.318193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.0.97.62:39295)
0929 15:48:02.319029 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:02.319042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:02.319085 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:48:02.320841 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:02.320865 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:02.321318 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:02.321742 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:02.323510 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:02.325013 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:02.326910 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:02.327650 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:02.327808 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:02.332905 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:02.332914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:02.332932 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:02.332988 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:02.336305 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:02.337144 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:02.337158 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:02.337165 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:02.337365 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:02.337814 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:02.337825 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:02.337828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:02.338436 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:02.338791 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:02.339248 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 15:48:02.339716 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":385,"threads_started":1}
I20260929 15:48:02.349145 14360 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:02.349582 14360 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:02.349696 14360 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b3b5a7abd64ff09a438d260b2bea52, State: Initialized, Role: FOLLOWER
I20260929 15:48:02.350268 14360 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:02.350520 14360 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:48:02.350658 14360 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:48:02.350832 14360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:02.353106 14360 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:02.353634 14360 leader_election.cc:302] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b3b5a7abd64ff09a438d260b2bea52; no voters:
I20260929 15:48:02.354270 14360 leader_election.cc:288] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:48:02.354425 14365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:02.355571 14365 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 LEADER]: Becoming Leader. State: Replica: e5b3b5a7abd64ff09a438d260b2bea52, State: Running, Role: LEADER
I20260929 15:48:02.356318 14365 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:02.356703 14360 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:48:02.359334 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b3b5a7abd64ff09a438d260b2bea52. Latest consensus state: current_term: 1 leader_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } } }
I20260929 15:48:02.359756 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:02.359570 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } } }
I20260929 15:48:02.359866 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:02.363587 14373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:48:02.368222 14373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:48:02.378057 14373 catalog_manager.cc:1347] Generated new cluster ID: a7dd12db8e834adcbebd91064573469f
I20260929 15:48:02.378208 14373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:48:02.399012 14373 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:48:02.400533 14373 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:48:02.422236 14373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Generated new TSK 0
I20260929 15:48:02.423189 14373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:48:02.492148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39295
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:02.856513 14386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:02.856871 14386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:02.857056 14386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:02.865885 14386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:02.866068 14386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:02.866248 14386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:48:02.877023 14386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:02.878495 14386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:02.879984 14386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:02.893785 14391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:02.893859 14392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:02.893810 14394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:02.894711 14386 server_base.cc:1034] running on GCE node
I20260929 15:48:02.895363 14386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:02.896435 14386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:02.897675 14386 hybrid_clock.cc:648] HybridClock initialized: now 1790696882897612 us; error 51 us; skew 500 ppm
I20260929 15:48:02.898015 14386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:02 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:02.901327 14386 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:48:02.902909 14386 webserver.cc:460] Webserver started at http://127.0.97.1:39675/ using document root <none> and password file <none>
I20260929 15:48:02.903642 14386 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:02.903801 14386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:02.904105 14386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:02.907434 14386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "fa6d7221ff564576a8c02638a00fd681"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "0f53433f2448a81bd549d9fdc73224e6"
server_key_iv: "bcb94edcbc169abc6113cc8ba09bc92d"
server_key_version: "encryptionkey@0"
I20260929 15:48:02.908422 14386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fa6d7221ff564576a8c02638a00fd681"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "0f53433f2448a81bd549d9fdc73224e6"
server_key_iv: "bcb94edcbc169abc6113cc8ba09bc92d"
server_key_version: "encryptionkey@0"
I20260929 15:48:02.914186 14386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:48:02.917796 14401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:02.919219 14386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:48:02.919401 14386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "fa6d7221ff564576a8c02638a00fd681"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "0f53433f2448a81bd549d9fdc73224e6"
server_key_iv: "bcb94edcbc169abc6113cc8ba09bc92d"
server_key_version: "encryptionkey@0"
I20260929 15:48:02.919620 14386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:02.950860 14386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:02.955626 14386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:02.955973 14386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:02.957147 14386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:02.958601 14386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:02.958710 14386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:02.958864 14386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:02.958937 14386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:02.994473 14386 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35423
I20260929 15:48:02.994519 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35423 every 8 connection(s)
I20260929 15:48:02.995850 14386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:48:03.000206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14386
I20260929 15:48:03.000434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:48:03.001286 388 external_mini_cluster.cc:1428] Setting key 257969150e628231ff63f3d7ed180ecc
Sep 29 15:48:03 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:03.020426 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:02.998674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47349 (local address 127.0.97.62:39295)
0929 15:48:02.999117 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:02.999122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:03.001996 (+ 2874us) server_negotiation.cc:407] Connection header received
0929 15:48:03.002955 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:03.002960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:03.003074 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:03.003225 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:03.004993 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.006170 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.007864 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.008643 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.008795 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:03.013013 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:03.013023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:03.013031 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:03.013084 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:03.016853 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:03.017685 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:03.017694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:03.017701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:03.017880 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:03.018312 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:03.018320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:03.018326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:03.018730 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:03.018899 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:03.019621 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 15:48:03.019909 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260929 15:48:03.021215 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:02.999272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:47349)
0929 15:48:03.000233 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:03.000286 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:03.002603 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:03.003528 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:03.003541 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:03.004128 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:03.004793 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:03.004812 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.006362 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.006366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:03.007677 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:03.007691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.008926 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.008932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:03.009176 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:03.009993 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:03.010024 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:03.012709 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:03.017039 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:03.017047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:03.017063 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:03.017530 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:03.017996 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:03.017999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:03.018001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:03.018182 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:03.018861 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:03.018867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:03.019215 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 15:48:03.019452 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 15:48:03.019881 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":380,"threads_started":1}
I20260929 15:48:03.022985 14514 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39295
I20260929 15:48:03.023361 14514 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:03.024413 14514 heartbeater.cc:511] Master 127.0.97.62:39295 requested a full tablet report, sending...
I20260929 15:48:03.027232 14325 ts_manager.cc:194] Registered new tserver with Master: fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:48:03.029224 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:48:03.058923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39295
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:03.396191 14520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:03.396503 14520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:03.396601 14520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:03.404106 14520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:03.404322 14520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:03.404546 14520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:48:03.414927 14520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:03.416420 14520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:03.417924 14520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:03.432157 14526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:03.432250 14525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:03.432199 14528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:03.433022 14520 server_base.cc:1034] running on GCE node
I20260929 15:48:03.433733 14520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:03.434782 14520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:03.436017 14520 hybrid_clock.cc:648] HybridClock initialized: now 1790696883435971 us; error 48 us; skew 500 ppm
I20260929 15:48:03.436401 14520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:03 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:03.439814 14520 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:48:03.441529 14520 webserver.cc:460] Webserver started at http://127.0.97.2:38269/ using document root <none> and password file <none>
I20260929 15:48:03.442298 14520 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:03.442446 14520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:03.442761 14520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:03.445824 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "822ab64c8a8443088fcd4b91b10a41cd"
format_stamp: "Formatted at 2026-09-29 15:48:03 on dist-test-slave-7491"
server_key: "df7540d6020b2130557b4d92fa27a8c6"
server_key_iv: "6872f9db4895c2692bdd225313c21945"
server_key_version: "encryptionkey@0"
I20260929 15:48:03.446628 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "822ab64c8a8443088fcd4b91b10a41cd"
format_stamp: "Formatted at 2026-09-29 15:48:03 on dist-test-slave-7491"
server_key: "df7540d6020b2130557b4d92fa27a8c6"
server_key_iv: "6872f9db4895c2692bdd225313c21945"
server_key_version: "encryptionkey@0"
I20260929 15:48:03.452303 14520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 15:48:03.456265 14535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:03.458056 14520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:48:03.458292 14520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "822ab64c8a8443088fcd4b91b10a41cd"
format_stamp: "Formatted at 2026-09-29 15:48:03 on dist-test-slave-7491"
server_key: "df7540d6020b2130557b4d92fa27a8c6"
server_key_iv: "6872f9db4895c2692bdd225313c21945"
server_key_version: "encryptionkey@0"
I20260929 15:48:03.458527 14520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:03.489490 14520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:03.494300 14520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:03.494621 14520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:03.495651 14520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:03.497185 14520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:03.497293 14520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:03.497433 14520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:03.497509 14520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:03.532949 14520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37397
I20260929 15:48:03.533042 14647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37397 every 8 connection(s)
I20260929 15:48:03.534307 14520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:48:03.543771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14520
I20260929 15:48:03.543953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:48:03.544459 388 external_mini_cluster.cc:1428] Setting key f55f6afc28210b1a7f5167b8d00d82ec
Sep 29 15:48:03 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:03.556308 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:03.537115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51001 (local address 127.0.97.62:39295)
0929 15:48:03.537563 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:03.537568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:03.538753 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 15:48:03.540748 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:03.540754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:03.540853 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:03.540976 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:03.542580 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.543326 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.545318 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.545836 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.545983 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:03.550133 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:03.550138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:03.550142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:03.550180 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:03.552947 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:03.553799 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:03.553804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:03.553806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:03.553940 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:03.554422 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:03.554426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:03.554428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:03.554710 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:03.554868 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:03.555615 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 15:48:03.555857 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260929 15:48:03.557416 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:03.537698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:51001)
0929 15:48:03.538496 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:03.538538 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:03.540457 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:03.541238 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:03.541246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:03.541797 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:03.542386 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:03.542403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.543496 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.543500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:03.545129 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:03.545144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:03.546101 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:03.546109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:03.546398 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:03.547167 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:03.547198 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:03.549853 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:03.553143 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:03.553155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:03.553181 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:03.553644 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:03.554061 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:03.554069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:03.554074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:03.554293 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:03.554840 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:03.554850 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:03.555227 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 15:48:03.555515 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 15:48:03.556098 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":341,"threads_started":1}
I20260929 15:48:03.559152 14648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39295
I20260929 15:48:03.559518 14648 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:03.560491 14648 heartbeater.cc:511] Master 127.0.97.62:39295 requested a full tablet report, sending...
I20260929 15:48:03.562448 14325 ts_manager.cc:194] Registered new tserver with Master: 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397)
I20260929 15:48:03.563402 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51001
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:48:03.600517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39295
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:03.958546 14654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:03.959034 14654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:03.959419 14654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:03.967549 14654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:03.967684 14654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:03.967854 14654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:48:03.978463 14654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:03.979831 14654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:03.981204 14654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:03.994591 14660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:03.994632 14659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:03.996835 14662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:04.000918 14654 server_base.cc:1034] running on GCE node
I20260929 15:48:04.001617 14654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:04.002646 14654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:04.003875 14654 hybrid_clock.cc:648] HybridClock initialized: now 1790696884003825 us; error 38 us; skew 500 ppm
I20260929 15:48:04.004295 14654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:04 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:04.007709 14654 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:48:04.009509 14654 webserver.cc:460] Webserver started at http://127.0.97.3:45379/ using document root <none> and password file <none>
I20260929 15:48:04.010300 14654 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:04.010437 14654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:04.010766 14654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:04.013933 14654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6b723a663744ca3a5061421f357193e"
format_stamp: "Formatted at 2026-09-29 15:48:04 on dist-test-slave-7491"
server_key: "047115d564e07dd9914e4be5ad0f670e"
server_key_iv: "5aa3cc6cc9c5e1c501d1d992e536a8ea"
server_key_version: "encryptionkey@0"
I20260929 15:48:04.014787 14654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6b723a663744ca3a5061421f357193e"
format_stamp: "Formatted at 2026-09-29 15:48:04 on dist-test-slave-7491"
server_key: "047115d564e07dd9914e4be5ad0f670e"
server_key_iv: "5aa3cc6cc9c5e1c501d1d992e536a8ea"
server_key_version: "encryptionkey@0"
I20260929 15:48:04.020490 14654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 15:48:04.024294 14669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:04.025645 14654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:48:04.025830 14654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "d6b723a663744ca3a5061421f357193e"
format_stamp: "Formatted at 2026-09-29 15:48:04 on dist-test-slave-7491"
server_key: "047115d564e07dd9914e4be5ad0f670e"
server_key_iv: "5aa3cc6cc9c5e1c501d1d992e536a8ea"
server_key_version: "encryptionkey@0"
I20260929 15:48:04.026031 14654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:04.032619 14514 heartbeater.cc:503] Master 127.0.97.62:39295 was elected leader, sending a full tablet report...
I20260929 15:48:04.040361 14654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:04.045116 14654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:04.045406 14654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:04.046388 14654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:04.047919 14654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:04.048038 14654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:04.048246 14654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:04.048327 14654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 15:48:04.083493 14654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44519
I20260929 15:48:04.083555 14781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44519 every 8 connection(s)
I20260929 15:48:04.084887 14654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:48:04.089159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14654
I20260929 15:48:04.089355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:48:04.089958 388 external_mini_cluster.cc:1428] Setting key 2e5b3fff4eca57f3bb6461cf87254d24
Sep 29 15:48:04 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:04.108212 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.087621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60633 (local address 127.0.97.62:39295)
0929 15:48:04.088331 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.088336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.089413 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 15:48:04.091995 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.092004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.092223 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.092378 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:04.094568 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.095364 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.097257 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.097684 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.097796 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.101625 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:04.101630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:04.101633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:04.101694 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:04.104754 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.105652 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.105656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.105665 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.105801 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.106282 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.106286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.106288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.106563 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:04.106731 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.107494 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.107763 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":441,"threads_started":1}
I20260929 15:48:04.109215 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.088156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:60633)
0929 15:48:04.089192 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:04.089259 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:04.091592 (+ 2333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:04.092741 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:04.092754 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:04.093529 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:04.094342 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.094367 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.095591 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.095596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.097067 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.097084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.097891 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.097895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.098103 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.098876 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:04.098901 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:04.101346 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:04.104952 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:04.104963 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:04.104988 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:04.105502 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:04.105924 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:04.105929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:04.105933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:04.106143 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:04.106696 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:04.106705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:04.107087 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 15:48:04.107377 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 15:48:04.107983 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":370,"threads_started":1}
I20260929 15:48:04.110809 14782 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39295
I20260929 15:48:04.111155 14782 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:04.112087 14782 heartbeater.cc:511] Master 127.0.97.62:39295 requested a full tablet report, sending...
I20260929 15:48:04.113885 14325 ts_manager.cc:194] Registered new tserver with Master: d6b723a663744ca3a5061421f357193e (127.0.97.3:44519)
I20260929 15:48:04.114773 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60633
I20260929 15:48:04.118618 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:48:04.136792 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.123465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.97.62:39295)
0929 15:48:04.123703 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.123708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.124111 (+ 403us) server_negotiation.cc:407] Connection header received
0929 15:48:04.124348 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.124353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.124435 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.124521 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:04.126015 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.126904 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.128565 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.128972 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.129076 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.131642 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:04.131647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:04.131651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:04.131690 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:04.134296 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.135023 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.135027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.135029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.135159 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.135494 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.135497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.135499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.135830 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:04.136007 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.136356 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.136536 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260929 15:48:04.141063 14325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:48:04.148633 14325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:48:04.176611 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.164292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37397 (local address 127.0.0.1:60086)
0929 15:48:04.168631 (+ 4339us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:04.168648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:04.168769 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:04.169415 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:04.169419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:04.169442 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:04.170126 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.170140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.171929 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.171936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.173644 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.173658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.175525 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.175534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.175728 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.175796 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 15:48:04.175952 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 15:48:04.176196 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4225,"thread_start_us":2884,"threads_started":1}
I20260929 15:48:04.176705 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.165899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.0.97.2:37397)
0929 15:48:04.166619 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.166632 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.168915 (+ 2283us) server_negotiation.cc:407] Connection header received
0929 15:48:04.169024 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.169034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.169224 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.169418 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:04.170325 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.171786 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.173827 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.175046 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.175170 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.175389 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.176044 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.176291 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":894,"threads_started":1}
I20260929 15:48:04.177424 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.162211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44519 (local address 127.0.0.1:41594)
0929 15:48:04.167476 (+ 5265us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:04.167502 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:04.167674 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:04.168678 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:04.168698 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:04.168731 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:04.169278 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.169295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.171322 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.171325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.174372 (+ 3047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.174390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.176513 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.176520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.176710 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.176751 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:48:04.176853 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:48:04.176977 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5106,"mutex_wait_us":142,"thread_start_us":1321,"threads_started":1}
I20260929 15:48:04.177973 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.162605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.97.3:44519)
0929 15:48:04.163559 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.163571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.167640 (+ 4069us) server_negotiation.cc:407] Connection header received
0929 15:48:04.167889 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.167903 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.168303 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.168535 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:04.169550 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.171124 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.174563 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.176006 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.176126 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.176370 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.177403 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.177598 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":500,"threads_started":1}
I20260929 15:48:04.178282 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.161432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35423 (local address 127.0.0.1:55472)
0929 15:48:04.164052 (+ 2620us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:04.164103 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:04.164367 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:04.168960 (+ 4593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:04.168964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:04.168984 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:04.169439 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.169452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.171548 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.171552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.175072 (+ 3520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.175089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.177336 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.177343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.177583 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.177625 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 15:48:04.177730 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:48:04.177879 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2284,"mutex_wait_us":112,"thread_start_us":6953,"threads_started":1}
I20260929 15:48:04.179191 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.161576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.0.97.1:35423)
0929 15:48:04.168382 (+ 6806us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.168393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.168423 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:48:04.168525 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.168532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.168768 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.168951 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:04.169702 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.171377 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.175376 (+ 3999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.176770 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.176932 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.177132 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.177962 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.178155 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6624,"thread_start_us":6426,"threads_started":1}
I20260929 15:48:04.181505 14583 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 (DEFAULT_TABLE table=test-table [id=4878b1894ce5471e910f61e3585ca291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:04.181743 14717 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 (DEFAULT_TABLE table=test-table [id=4878b1894ce5471e910f61e3585ca291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:04.182803 14583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:04.183084 14717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:04.182796 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 (DEFAULT_TABLE table=test-table [id=4878b1894ce5471e910f61e3585ca291]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:04.184350 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:04.194888 14801 tablet_bootstrap.cc:492] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Bootstrap starting.
I20260929 15:48:04.197968 14802 tablet_bootstrap.cc:492] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Bootstrap starting.
I20260929 15:48:04.198588 14801 tablet_bootstrap.cc:654] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:04.200043 14801 log.cc:824] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:04.200263 14803 tablet_bootstrap.cc:492] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Bootstrap starting.
I20260929 15:48:04.202332 14802 tablet_bootstrap.cc:654] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:04.203496 14803 tablet_bootstrap.cc:654] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:04.204115 14802 log.cc:824] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:04.204622 14803 log.cc:824] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:04.207597 14803 tablet_bootstrap.cc:492] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: No bootstrap required, opened a new log
I20260929 15:48:04.208016 14803 ts_tablet_manager.cc:1397] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 15:48:04.209518 14801 tablet_bootstrap.cc:492] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: No bootstrap required, opened a new log
I20260929 15:48:04.209816 14802 tablet_bootstrap.cc:492] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: No bootstrap required, opened a new log
I20260929 15:48:04.209822 14801 ts_tablet_manager.cc:1397] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260929 15:48:04.210067 14802 ts_tablet_manager.cc:1397] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 15:48:04.216463 14803 raft_consensus.cc:348] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.216929 14803 raft_consensus.cc:374] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:04.217054 14803 raft_consensus.cc:729] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6d7221ff564576a8c02638a00fd681, State: Initialized, Role: FOLLOWER
I20260929 15:48:04.217715 14803 consensus_queue.cc:260] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.219640 14803 ts_tablet_manager.cc:1428] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 15:48:04.221016 14802 raft_consensus.cc:348] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.221016 14801 raft_consensus.cc:348] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.221495 14801 raft_consensus.cc:374] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:04.221495 14802 raft_consensus.cc:374] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:04.221603 14802 raft_consensus.cc:729] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b723a663744ca3a5061421f357193e, State: Initialized, Role: FOLLOWER
I20260929 15:48:04.221603 14801 raft_consensus.cc:729] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ab64c8a8443088fcd4b91b10a41cd, State: Initialized, Role: FOLLOWER
I20260929 15:48:04.222247 14801 consensus_queue.cc:260] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.222266 14802 consensus_queue.cc:260] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.224511 14648 heartbeater.cc:503] Master 127.0.97.62:39295 was elected leader, sending a full tablet report...
I20260929 15:48:04.224615 14802 ts_tablet_manager.cc:1428] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 15:48:04.225256 14801 ts_tablet_manager.cc:1428] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 15:48:04.226481 14782 heartbeater.cc:503] Master 127.0.97.62:39295 was elected leader, sending a full tablet report...
W20260929 15:48:04.248646 14515 tablet.cc:2369] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:48:04.286517 14649 tablet.cc:2369] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:48:04.294911 14809 raft_consensus.cc:482] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:48:04.295212 14809 raft_consensus.cc:504] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.297137 14809 leader_election.cc:288] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423), 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397)
I20260929 15:48:04.304571 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.297625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53415 (local address 127.0.97.2:37397)
0929 15:48:04.297842 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.297850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.298069 (+ 219us) server_negotiation.cc:407] Connection header received
0929 15:48:04.298300 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.298304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.298400 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.298491 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:04.299011 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.299949 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.302468 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.303595 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.303750 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.303859 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.304121 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.304312 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:48:04.304603 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.297626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37397 (local address 127.0.97.3:53415)
0929 15:48:04.297962 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:04.297980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:04.298148 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:04.298557 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:04.298569 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:04.298608 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:04.298847 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.298862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.300140 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.300145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.302249 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.302267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.303787 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.303792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.303946 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.303973 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 15:48:04.304061 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 15:48:04.304240 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":26}
I20260929 15:48:04.305260 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.297649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60483 (local address 127.0.97.1:35423)
0929 15:48:04.297887 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.297893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.298749 (+ 856us) server_negotiation.cc:407] Connection header received
0929 15:48:04.299007 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.299012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.299100 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.299222 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:04.299765 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.300798 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.302983 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.304214 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.304351 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.304452 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.304831 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.304991 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:48:04.305773 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.297741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35423 (local address 127.0.97.3:60483)
0929 15:48:04.298618 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:04.298642 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:04.298781 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:04.299275 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:04.299279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:04.299304 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:04.299580 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.299587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.300949 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.300954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.302815 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:04.302825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.304440 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.304447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:04.304619 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.304660 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:48:04.305339 (+ 679us) client_negotiation.cc:240] Negotiation successful
0929 15:48:04.305484 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":49,"thread_start_us":482,"threads_started":1}
I20260929 15:48:04.305811 14603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f73f4e6cb4481ac3cd5e8f663a735" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ab64c8a8443088fcd4b91b10a41cd" is_pre_election: true
I20260929 15:48:04.306401 14603 raft_consensus.cc:2454] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 0.
I20260929 15:48:04.306825 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f73f4e6cb4481ac3cd5e8f663a735" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6d7221ff564576a8c02638a00fd681" is_pre_election: true
I20260929 15:48:04.307109 14672 leader_election.cc:302] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822ab64c8a8443088fcd4b91b10a41cd, d6b723a663744ca3a5061421f357193e; no voters:
I20260929 15:48:04.307369 14469 raft_consensus.cc:2454] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 0.
I20260929 15:48:04.307543 14809 raft_consensus.cc:2781] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:48:04.307717 14809 raft_consensus.cc:482] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:48:04.307840 14809 raft_consensus.cc:3037] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:04.310205 14809 raft_consensus.cc:504] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.310891 14809 leader_election.cc:288] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 election: Requested vote from peers fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423), 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397)
I20260929 15:48:04.311455 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f73f4e6cb4481ac3cd5e8f663a735" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6d7221ff564576a8c02638a00fd681"
I20260929 15:48:04.311532 14603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4f73f4e6cb4481ac3cd5e8f663a735" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ab64c8a8443088fcd4b91b10a41cd"
I20260929 15:48:04.311713 14469 raft_consensus.cc:3037] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:04.311769 14603 raft_consensus.cc:3037] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:04.314072 14469 raft_consensus.cc:2454] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 1.
I20260929 15:48:04.314574 14671 leader_election.cc:302] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b723a663744ca3a5061421f357193e, fa6d7221ff564576a8c02638a00fd681; no voters:
I20260929 15:48:04.314919 14809 raft_consensus.cc:2781] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:04.315021 14603 raft_consensus.cc:2454] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 1.
I20260929 15:48:04.315521 14809 raft_consensus.cc:686] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 1 LEADER]: Becoming Leader. State: Replica: d6b723a663744ca3a5061421f357193e, State: Running, Role: LEADER
I20260929 15:48:04.316560 14809 consensus_queue.cc:237] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } }
I20260929 15:48:04.322705 14324 catalog_manager.cc:5495] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b723a663744ca3a5061421f357193e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d6b723a663744ca3a5061421f357193e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } health_report { overall_health: UNKNOWN } } }
W20260929 15:48:04.336594 14783 tablet.cc:2369] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:48:04.370379 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.364939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.97.3:44519)
0929 15:48:04.365144 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.365152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.365177 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:48:04.365436 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.365440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.365522 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.365678 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:48:04.366051 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.366762 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.368132 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.368593 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.368734 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.369238 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:48:04.369746 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:48:04.369924 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.369983 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.370067 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:48:04.379195 14469 raft_consensus.cc:1261] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Refusing update from remote peer d6b723a663744ca3a5061421f357193e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:04.379274 14603 raft_consensus.cc:1261] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Refusing update from remote peer d6b723a663744ca3a5061421f357193e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:04.380326 14811 consensus_queue.cc:1035] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:04.380743 14809 consensus_queue.cc:1035] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [LEADER]: Connected to new peer: Peer: permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:04.400039 14815 mvcc.cc:205] Tried to move back new op lower bound from 7334694438405672960 to 7334694438165626880. Current Snapshot: MvccSnapshot[applied={T|T < 7334694438405672960}]
I20260929 15:48:04.411911 14324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:04.412236 14324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:04.415714 14324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Sending DeleteTablet for 3 replicas of tablet 4c4f73f4e6cb4481ac3cd5e8f663a735
I20260929 15:48:04.417006 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55472
I20260929 15:48:04.417277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14386
I20260929 15:48:04.417393 14717 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41594
I20260929 15:48:04.417790 14583 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60086
I20260929 15:48:04.433272 14824 tablet_replica.cc:330] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: stopping tablet replica
I20260929 15:48:04.433915 14824 raft_consensus.cc:2229] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:04.434594 14824 raft_consensus.cc:2258] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:04.435828 14827 tablet_replica.cc:330] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: stopping tablet replica
I20260929 15:48:04.436421 14827 raft_consensus.cc:2229] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:04.437050 14827 raft_consensus.cc:2258] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:04.439960 14827 ts_tablet_manager.cc:1905] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:04.439998 14824 ts_tablet_manager.cc:1905] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:04.447182 14827 ts_tablet_manager.cc:1918] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:04.448015 14825 tablet_replica.cc:330] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: stopping tablet replica
I20260929 15:48:04.448647 14825 raft_consensus.cc:2229] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:48:04.449608 14825 raft_consensus.cc:2258] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 15:48:04 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:48:04.450222 14827 log.cc:1196] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/4c4f73f4e6cb4481ac3cd5e8f663a735
I20260929 15:48:04.450786 14824 ts_tablet_manager.cc:1918] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:04.450966 14824 log.cc:1196] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/4c4f73f4e6cb4481ac3cd5e8f663a735
I20260929 15:48:04.451581 14824 ts_tablet_manager.cc:1939] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P fa6d7221ff564576a8c02638a00fd681: Deleting consensus metadata
I20260929 15:48:04.452654 14825 ts_tablet_manager.cc:1905] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:04.452736 14827 ts_tablet_manager.cc:1939] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P 822ab64c8a8443088fcd4b91b10a41cd: Deleting consensus metadata
I20260929 15:48:04.453799 14310 catalog_manager.cc:4841] TS fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423): tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 (table test-table [id=4878b1894ce5471e910f61e3585ca291]) successfully deleted
I20260929 15:48:04.454499 14311 catalog_manager.cc:4841] TS 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397): tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 (table test-table [id=4878b1894ce5471e910f61e3585ca291]) successfully deleted
I20260929 15:48:04.459714 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.418558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.97.1:35423)
0929 15:48:04.418782 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.418790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.429019 (+ 10229us) server_negotiation.cc:407] Connection header received
0929 15:48:04.429595 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.429605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.429750 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.429851 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:04.435100 (+ 5249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.436321 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.442952 (+ 6631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.443615 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.443742 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.451228 (+ 7486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:04.451245 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:04.451273 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:04.451325 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:04.455595 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.456734 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.456747 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.456750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.456955 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.457465 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.457470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.457473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.457979 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:04.458311 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.459017 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.459297 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:48:04.460572 14509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:04.461114 14825 ts_tablet_manager.cc:1918] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:04.461329 14825 log.cc:1196] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/4c4f73f4e6cb4481ac3cd5e8f663a735
I20260929 15:48:04.462045 14825 ts_tablet_manager.cc:1939] T 4c4f73f4e6cb4481ac3cd5e8f663a735 P d6b723a663744ca3a5061421f357193e: Deleting consensus metadata
I20260929 15:48:04.463888 14311 catalog_manager.cc:4841] TS d6b723a663744ca3a5061421f357193e (127.0.97.3:44519): tablet 4c4f73f4e6cb4481ac3cd5e8f663a735 (table test-table [id=4878b1894ce5471e910f61e3585ca291]) successfully deleted
I20260929 15:48:04.580296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14386
I20260929 15:48:04.602169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14520
Sep 29 15:48:04 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:48:04.626860 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.603035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.0.97.2:37397)
0929 15:48:04.603212 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.603216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.603236 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:48:04.603447 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.603451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.603533 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.603619 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:04.605397 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.606180 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.608308 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.608896 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.609005 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.618957 (+ 9952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:04.618983 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:04.619005 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:04.619067 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:04.623294 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.624344 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.624355 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.624360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.624543 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.625040 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.625046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.625050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.625585 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:04.625916 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.626192 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.626447 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:48:04.627689 14643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:04.731848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14520
I20260929 15:48:04.756848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14654
Sep 29 15:48:04 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:48:04.776537 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:04.757846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.97.3:44519)
0929 15:48:04.758046 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:04.758052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:04.758076 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:48:04.758156 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:04.758162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:04.758302 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:04.758428 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:04.759638 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.760574 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.762157 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:04.762704 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:04.762881 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:04.768662 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:04.768676 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:04.768691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:04.768742 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:04.772784 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.773685 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.773696 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.773699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.773873 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:04.774284 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:04.774289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:04.774299 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:04.774718 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:04.775381 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:04.775758 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 15:48:04.776030 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:48:04.777341 14777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:04.906814 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14654
I20260929 15:48:04.928664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14293
I20260929 15:48:04.929893 14355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:05.044780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14293
I20260929 15:48:05.067193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39295
--webserver_interface=127.0.97.62
--webserver_port=34479
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:05.419909 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:05.420318 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:05.420451 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:05.428049 14832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:48:05.428396 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:05.428464 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:05.428529 14832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:48:05.428573 14832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:48:05.440403 14832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:05.441859 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:05.443216 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:05.468755 14839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:05.468812 14838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:05.468847 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:05.469820 14832 server_base.cc:1034] running on GCE node
I20260929 15:48:05.471072 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:05.472927 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:05.474222 14832 hybrid_clock.cc:648] HybridClock initialized: now 1790696885474168 us; error 47 us; skew 500 ppm
I20260929 15:48:05.474704 14832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:05 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:05.484529 14832 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:48:05.486905 14832 webserver.cc:460] Webserver started at http://127.0.97.62:34479/ using document root <none> and password file <none>
I20260929 15:48:05.487856 14832 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:05.488041 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:05.495262 14832 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 15:48:05.499135 14848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:05.501396 14832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 15:48:05.501662 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "e5b3b5a7abd64ff09a438d260b2bea52"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "f248af9947876c958548590989b6a55e"
server_key_iv: "cbb4ad8f20f40991f4be0d91e081d3a1"
server_key_version: "encryptionkey@0"
I20260929 15:48:05.502965 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:05.530277 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:05.535543 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:05.535894 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:05.558151 14832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39295
I20260929 15:48:05.558130 14899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39295 every 8 connection(s)
I20260929 15:48:05.559954 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:48:05.563613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14832
I20260929 15:48:05.564697 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35423
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39295
--webserver_port=39675
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:48:05.570467 14900 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:48:05.575397 14900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap starting.
I20260929 15:48:05.596926 14900 log.cc:824] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:05.610392 14900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:48:05.611018 14900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap complete.
I20260929 15:48:05.620896 14900 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:05.621914 14900 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b3b5a7abd64ff09a438d260b2bea52, State: Initialized, Role: FOLLOWER
I20260929 15:48:05.622653 14900 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:05.622990 14900 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:48:05.623245 14900 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:48:05.623433 14900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:48:05.627655 14900 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:05.628357 14900 leader_election.cc:302] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b3b5a7abd64ff09a438d260b2bea52; no voters:
I20260929 15:48:05.629022 14900 leader_election.cc:288] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:48:05.629168 14904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:48:05.630324 14904 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 LEADER]: Becoming Leader. State: Replica: e5b3b5a7abd64ff09a438d260b2bea52, State: Running, Role: LEADER
I20260929 15:48:05.631475 14900 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:48:05.631806 14904 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:05.638015 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } } }
I20260929 15:48:05.638454 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:05.639165 14906 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b3b5a7abd64ff09a438d260b2bea52. Latest consensus state: current_term: 2 leader_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } } }
I20260929 15:48:05.639500 14906 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:05.649729 14911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:48:05.658231 14911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:48:05.659804 14911 catalog_manager.cc:1259] Loaded cluster ID: a7dd12db8e834adcbebd91064573469f
I20260929 15:48:05.659929 14911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:48:05.663234 14911 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 15:48:05.681196 14911 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 15:48:05.692373 14911 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:48:05.694360 14911 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Loaded TSK: 0
I20260929 15:48:05.696152 14911 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:48:05.947845 14902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:05.948297 14902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:05.948465 14902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:05.956874 14902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:05.957060 14902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:05.957259 14902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:48:05.968055 14902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39675
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:05.969583 14902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:05.971084 14902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:05.989123 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:05.989480 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:05.990129 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:05.990799 14902 server_base.cc:1034] running on GCE node
I20260929 15:48:05.991560 14902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:05.992712 14902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:05.994026 14902 hybrid_clock.cc:648] HybridClock initialized: now 1790696885993914 us; error 66 us; skew 500 ppm
I20260929 15:48:05.994433 14902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:05 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:05.998576 14902 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:48:06.000483 14902 webserver.cc:460] Webserver started at http://127.0.97.1:39675/ using document root <none> and password file <none>
I20260929 15:48:06.001261 14902 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:06.001443 14902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:06.007604 14902 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:48:06.010812 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.012509 14902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:48:06.012719 14902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "fa6d7221ff564576a8c02638a00fd681"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "0f53433f2448a81bd549d9fdc73224e6"
server_key_iv: "bcb94edcbc169abc6113cc8ba09bc92d"
server_key_version: "encryptionkey@0"
I20260929 15:48:06.013818 14902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:06.029917 14902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:06.035041 14902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:06.035346 14902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:06.036506 14902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:06.037954 14902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:06.038057 14902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.038204 14902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:06.038295 14902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.077394 14902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35423
I20260929 15:48:06.077481 15050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35423 every 8 connection(s)
I20260929 15:48:06.079041 14902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:48:06.083573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 14902
I20260929 15:48:06.084410 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37397
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39295
--webserver_port=38269
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:48:06 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:06.107824 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:06.082239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38059 (local address 127.0.97.62:39295)
0929 15:48:06.083013 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:06.083034 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:06.083725 (+ 691us) server_negotiation.cc:407] Connection header received
0929 15:48:06.087124 (+ 3399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:06.087138 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:06.087556 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:06.088032 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:06.090372 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.091680 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.093428 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.094107 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.094337 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:06.099030 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:06.099043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:06.099064 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:06.099126 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:06.102457 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:06.103693 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:06.103701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:06.103706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:06.103889 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:06.104522 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:06.104526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:06.104528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:06.105092 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:06.105254 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:06.106312 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0929 15:48:06.106715 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":276,"threads_started":1}
I20260929 15:48:06.108204 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:06.082458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:38059)
0929 15:48:06.083485 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:06.083540 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:06.086421 (+ 2881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:06.088294 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:06.088307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:06.089228 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:06.090066 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:06.090100 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.091887 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.091893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:06.093254 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:06.093273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.094373 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.094380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:06.094654 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:06.095389 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:06.095414 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:06.098526 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:06.102676 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:06.102690 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:06.102711 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:06.103492 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:06.104022 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:06.104030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:06.104036 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:06.104357 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:06.105251 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:06.105261 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:06.105734 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 15:48:06.106088 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 15:48:06.106754 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":429,"threads_started":1}
I20260929 15:48:06.111373 15051 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39295
I20260929 15:48:06.111735 15051 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:06.112628 15051 heartbeater.cc:511] Master 127.0.97.62:39295 requested a full tablet report, sending...
I20260929 15:48:06.114987 14865 ts_manager.cc:194] Registered new tserver with Master: fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423)
I20260929 15:48:06.116582 14865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38059
W20260929 15:48:06.433490 15055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:06.433852 15055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:06.434022 15055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:06.441990 15055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:06.442194 15055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:06.442390 15055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:48:06.452770 15055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38269
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:06.454205 15055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:06.455557 15055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:06.469267 15060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:06.469373 15063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:06.469292 15061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:06.470438 15055 server_base.cc:1034] running on GCE node
I20260929 15:48:06.471077 15055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:06.472400 15055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:06.473645 15055 hybrid_clock.cc:648] HybridClock initialized: now 1790696886473587 us; error 51 us; skew 500 ppm
I20260929 15:48:06.473999 15055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:06 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:06.477435 15055 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:48:06.479070 15055 webserver.cc:460] Webserver started at http://127.0.97.2:38269/ using document root <none> and password file <none>
I20260929 15:48:06.479875 15055 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:06.480047 15055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:06.485713 15055 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 15:48:06.488979 15070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.490661 15055 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:48:06.490862 15055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "822ab64c8a8443088fcd4b91b10a41cd"
format_stamp: "Formatted at 2026-09-29 15:48:03 on dist-test-slave-7491"
server_key: "df7540d6020b2130557b4d92fa27a8c6"
server_key_iv: "6872f9db4895c2692bdd225313c21945"
server_key_version: "encryptionkey@0"
I20260929 15:48:06.491894 15055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:06.505148 15055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:06.510026 15055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:06.510368 15055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:06.511446 15055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:06.513028 15055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:06.513161 15055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.513293 15055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:06.513401 15055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.547642 15055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37397
I20260929 15:48:06.547837 15182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37397 every 8 connection(s)
I20260929 15:48:06.549237 15055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:48:06.551101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15055
I20260929 15:48:06.551801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44519
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39295
--webserver_port=45379
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:48:06 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:06.575459 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:06.552350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49623 (local address 127.0.97.62:39295)
0929 15:48:06.552692 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:06.552702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:06.554040 (+ 1338us) server_negotiation.cc:407] Connection header received
0929 15:48:06.556037 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:06.556047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:06.556191 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:06.556313 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:06.558042 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.558987 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.561029 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.561564 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.561714 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:06.566737 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:06.566748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:06.566756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:06.566807 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:06.570858 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:06.571991 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:06.572001 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:06.572007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:06.572262 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:06.572809 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:06.572818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:06.572826 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:06.573308 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:06.573481 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:06.574476 (+ 995us) server_negotiation.cc:299] Negotiation successful
0929 15:48:06.574809 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 15:48:06.576390 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:06.552902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:49623)
0929 15:48:06.553827 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:06.553877 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:06.555687 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:06.556567 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:06.556576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:06.557159 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:06.557845 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:06.557865 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.559229 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.559235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:06.560868 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:06.560882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:06.561814 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:06.561824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:06.562163 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:06.563053 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:06.563090 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:06.566415 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:06.571070 (+ 4655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:06.571081 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:06.571098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:06.571807 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:06.572395 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:06.572400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:06.572404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:06.572661 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:06.573485 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:06.573493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:06.573941 (+ 448us) client_negotiation.cc:769] Sending connection context
0929 15:48:06.574275 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 15:48:06.574851 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":408,"threads_started":1}
I20260929 15:48:06.578037 15183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39295
I20260929 15:48:06.578354 15183 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:06.579078 15183 heartbeater.cc:511] Master 127.0.97.62:39295 requested a full tablet report, sending...
I20260929 15:48:06.580703 14865 ts_manager.cc:194] Registered new tserver with Master: 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397)
I20260929 15:48:06.581568 14865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49623
W20260929 15:48:06.910900 15186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:06.911262 15186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:06.911440 15186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:06.919023 15186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:06.919193 15186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:06.919381 15186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:48:06.929731 15186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45379
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:06.931124 15186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:06.932628 15186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:06.946596 15194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:06.947113 15192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:06.947317 15191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:06.948148 15186 server_base.cc:1034] running on GCE node
I20260929 15:48:06.948923 15186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:06.950083 15186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:06.951345 15186 hybrid_clock.cc:648] HybridClock initialized: now 1790696886951295 us; error 50 us; skew 500 ppm
I20260929 15:48:06.951722 15186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:06 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:06.955169 15186 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:48:06.956913 15186 webserver.cc:460] Webserver started at http://127.0.97.3:45379/ using document root <none> and password file <none>
I20260929 15:48:06.957681 15186 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:06.957854 15186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:06.963729 15186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:48:06.966842 15201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.968489 15186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:48:06.968700 15186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "d6b723a663744ca3a5061421f357193e"
format_stamp: "Formatted at 2026-09-29 15:48:04 on dist-test-slave-7491"
server_key: "047115d564e07dd9914e4be5ad0f670e"
server_key_iv: "5aa3cc6cc9c5e1c501d1d992e536a8ea"
server_key_version: "encryptionkey@0"
I20260929 15:48:06.969784 15186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:06.985724 15186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:06.990976 15186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:06.991297 15186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:06.992419 15186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:06.993793 15186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:06.993919 15186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:06.994048 15186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:06.994151 15186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:07.027776 15186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44519
I20260929 15:48:07.027848 15313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44519 every 8 connection(s)
I20260929 15:48:07.029397 15186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:48:07.032209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15186
Sep 29 15:48:07 dist-test-slave-7491 krb5kdc[14280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:07.050387 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.033189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.0.97.62:39295)
0929 15:48:07.033749 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.033758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.035069 (+ 1311us) server_negotiation.cc:407] Connection header received
0929 15:48:07.035246 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.035250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.035323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.035417 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:07.037528 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.038254 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.039921 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.040435 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.040573 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.044049 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:07.044054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:07.044058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:07.044089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:07.047409 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.048295 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.048302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.048309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.048444 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.048833 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.048841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.048848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.049186 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:07.049459 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.049717 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.049997 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":289,"threads_started":1}
I20260929 15:48:07.052838 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.032339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34175 (local address 127.0.97.62:39295)
0929 15:48:07.032950 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.032957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.034303 (+ 1346us) server_negotiation.cc:407] Connection header received
0929 15:48:07.037184 (+ 2881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.037191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.037288 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.037374 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:07.039219 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.039924 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.041699 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.042396 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.042547 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.046113 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:07.046118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:07.046121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:07.046163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:07.049018 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.050033 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.050045 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.050049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.050206 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.050743 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.050746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.050748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.051120 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:07.051261 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.052280 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.052491 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458}
I20260929 15:48:07.053790 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.032961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:34175)
0929 15:48:07.034117 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:07.034173 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:07.036836 (+ 2663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:07.037666 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:07.037675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:07.038188 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:07.039016 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.039044 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.040107 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.040115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.041505 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.041518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.042608 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.042613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.042820 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.043506 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:07.043528 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:07.045881 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:07.049178 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:07.049188 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:07.049209 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:07.049885 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:07.050327 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:07.050334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:07.050339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:07.050612 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:07.051261 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:07.051270 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:07.051758 (+ 488us) client_negotiation.cc:769] Sending connection context
0929 15:48:07.052097 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 15:48:07.052579 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":611,"threads_started":1}
I20260929 15:48:07.055069 15314 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39295
I20260929 15:48:07.055384 15314 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:07.056102 15314 heartbeater.cc:511] Master 127.0.97.62:39295 requested a full tablet report, sending...
I20260929 15:48:07.057533 14865 ts_manager.cc:194] Registered new tserver with Master: d6b723a663744ca3a5061421f357193e (127.0.97.3:44519)
I20260929 15:48:07.058353 14865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34175
I20260929 15:48:07.066196 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:48:07.073257 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.067088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.0.97.62:39295)
0929 15:48:07.067292 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.067300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.067821 (+ 521us) server_negotiation.cc:407] Connection header received
0929 15:48:07.067957 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.067961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.068123 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.068323 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:48:07.068773 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.069491 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.070987 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.071431 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.071537 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.071976 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:48:07.072595 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:48:07.072712 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.072852 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.072925 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:48:07.074224 14865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:48:07.082018 14865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:48:07.108276 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.094591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35423 (local address 127.0.0.1:55490)
0929 15:48:07.095589 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:07.095638 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:07.095851 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:07.097270 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:07.097280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:07.097321 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:07.097822 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.097840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.101270 (+ 3430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.101280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.104906 (+ 3626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.104944 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.107178 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.107184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.107397 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.107448 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:48:07.107592 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 15:48:07.107815 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":688,"threads_started":1}
I20260929 15:48:07.108331 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.095036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.0.97.1:35423)
0929 15:48:07.096609 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.096621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.096652 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:48:07.096753 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.096763 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.097025 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.097254 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:07.098048 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.101049 (+ 3001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.105157 (+ 4108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.106645 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.106800 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.107036 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.107680 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.107913 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":1267,"threads_started":1}
I20260929 15:48:07.110595 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.094947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37397 (local address 127.0.0.1:60116)
0929 15:48:07.097404 (+ 2457us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:07.097428 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:07.097550 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:07.098688 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:07.098697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:07.098735 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:07.099239 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.099255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.102668 (+ 3413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.102677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.106899 (+ 4222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.106911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.109760 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.109767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.109955 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.109992 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:48:07.110101 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:48:07.110253 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2317,"thread_start_us":2161,"threads_started":1}
I20260929 15:48:07.110677 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.095142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.0.97.2:37397)
0929 15:48:07.096993 (+ 1851us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.097001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.097540 (+ 539us) server_negotiation.cc:407] Connection header received
0929 15:48:07.097788 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.097799 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.098176 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.098385 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:07.100369 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.102435 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.107068 (+ 4633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.109213 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.109346 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.109564 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.110173 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.110368 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1701,"thread_start_us":1606,"threads_started":1}
I20260929 15:48:07.112815 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.097518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44519 (local address 127.0.0.1:41628)
0929 15:48:07.104704 (+ 7186us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:07.104724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:07.104873 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:07.105598 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:07.105603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:07.105630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:07.106085 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.106098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.107933 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.107939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.110182 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.110196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.111977 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.111984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.112201 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.112240 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:48:07.112350 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 15:48:07.112478 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7029,"thread_start_us":6869,"threads_started":1}
I20260929 15:48:07.112881 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.098040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.0.97.3:44519)
0929 15:48:07.103532 (+ 5492us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.103539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.104861 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 15:48:07.105064 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.105074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.105372 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.105588 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:07.106274 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.107718 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.110371 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.111752 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.111908 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.112133 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.112400 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.112551 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5325,"thread_start_us":5158,"threads_started":1}
I20260929 15:48:07.112761 14986 tablet_service.cc:1467] Processing CreateTablet for tablet 6fda3cb244e74b4cb65115a60bef039d (DEFAULT_TABLE table=test-table [id=f90ef5fc807747e8a05cf9074410f97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:07.114351 14986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fda3cb244e74b4cb65115a60bef039d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:07.114377 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 6fda3cb244e74b4cb65115a60bef039d (DEFAULT_TABLE table=test-table [id=f90ef5fc807747e8a05cf9074410f97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:07.115899 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fda3cb244e74b4cb65115a60bef039d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:07.116230 15249 tablet_service.cc:1467] Processing CreateTablet for tablet 6fda3cb244e74b4cb65115a60bef039d (DEFAULT_TABLE table=test-table [id=f90ef5fc807747e8a05cf9074410f97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:07.117314 15249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fda3cb244e74b4cb65115a60bef039d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:07.120630 15051 heartbeater.cc:503] Master 127.0.97.62:39295 was elected leader, sending a full tablet report...
I20260929 15:48:07.130678 15330 tablet_bootstrap.cc:492] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Bootstrap starting.
I20260929 15:48:07.132874 15332 tablet_bootstrap.cc:492] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Bootstrap starting.
I20260929 15:48:07.133754 15331 tablet_bootstrap.cc:492] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Bootstrap starting.
I20260929 15:48:07.134853 15330 tablet_bootstrap.cc:654] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:07.136278 15330 log.cc:824] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:07.137043 15332 tablet_bootstrap.cc:654] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:07.137442 15331 tablet_bootstrap.cc:654] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:07.138314 15332 log.cc:824] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:07.139245 15331 log.cc:824] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:07.141774 15332 tablet_bootstrap.cc:492] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: No bootstrap required, opened a new log
I20260929 15:48:07.142107 15332 ts_tablet_manager.cc:1397] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 15:48:07.142443 15331 tablet_bootstrap.cc:492] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: No bootstrap required, opened a new log
I20260929 15:48:07.142733 15330 tablet_bootstrap.cc:492] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: No bootstrap required, opened a new log
I20260929 15:48:07.142727 15331 ts_tablet_manager.cc:1397] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:48:07.142993 15330 ts_tablet_manager.cc:1397] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260929 15:48:07.150987 15332 raft_consensus.cc:348] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.151467 15332 raft_consensus.cc:374] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:07.151587 15332 raft_consensus.cc:729] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6d7221ff564576a8c02638a00fd681, State: Initialized, Role: FOLLOWER
I20260929 15:48:07.152359 15332 consensus_queue.cc:260] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.154165 15332 ts_tablet_manager.cc:1428] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260929 15:48:07.154371 15330 raft_consensus.cc:348] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.154856 15330 raft_consensus.cc:374] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:07.154968 15330 raft_consensus.cc:729] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b723a663744ca3a5061421f357193e, State: Initialized, Role: FOLLOWER
I20260929 15:48:07.155020 15331 raft_consensus.cc:348] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.155514 15331 raft_consensus.cc:374] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:07.155622 15331 raft_consensus.cc:729] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822ab64c8a8443088fcd4b91b10a41cd, State: Initialized, Role: FOLLOWER
I20260929 15:48:07.155694 15330 consensus_queue.cc:260] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.156226 15331 consensus_queue.cc:260] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.157088 15314 heartbeater.cc:503] Master 127.0.97.62:39295 was elected leader, sending a full tablet report...
I20260929 15:48:07.157761 15330 ts_tablet_manager.cc:1428] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 15:48:07.159519 15183 heartbeater.cc:503] Master 127.0.97.62:39295 was elected leader, sending a full tablet report...
I20260929 15:48:07.159626 15331 ts_tablet_manager.cc:1428] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260929 15:48:07.226521 15337 raft_consensus.cc:482] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:48:07.226872 15337 raft_consensus.cc:504] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.228790 15337 leader_election.cc:288] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423), 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397)
I20260929 15:48:07.236052 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.229242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46895 (local address 127.0.97.1:35423)
0929 15:48:07.229477 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.229485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.229509 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:48:07.229853 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.229862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.229957 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.230056 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:07.230645 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.231648 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.234157 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.234990 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.235166 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.235291 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.235597 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.235741 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:48:07.236063 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.229017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35423 (local address 127.0.97.3:46895)
0929 15:48:07.229426 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:07.229447 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:07.229641 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:07.230143 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:07.230148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:07.230181 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:07.230435 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.230444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.231823 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.231830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.233950 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.233963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.235183 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.235190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.235397 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.235426 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:48:07.235554 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:48:07.235708 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 15:48:07.237581 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.229414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37115 (local address 127.0.97.2:37397)
0929 15:48:07.229713 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.229717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.231082 (+ 1365us) server_negotiation.cc:407] Connection header received
0929 15:48:07.231177 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.231183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.231269 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.231373 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:07.232336 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.233471 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.235236 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.236124 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.236302 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.236404 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.237101 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.237269 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:48:07.237998 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.230021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37397 (local address 127.0.97.3:37115)
0929 15:48:07.230953 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:07.230969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:07.231076 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:07.231578 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:07.231582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:07.231605 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:07.232076 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.232088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.233682 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.233686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.235027 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:07.235039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.236566 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.236572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:07.236773 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.236823 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 15:48:07.236992 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 15:48:07.237156 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":475,"threads_started":1}
I20260929 15:48:07.237648 15006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda3cb244e74b4cb65115a60bef039d" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6d7221ff564576a8c02638a00fd681" is_pre_election: true
I20260929 15:48:07.238639 15006 raft_consensus.cc:2454] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 0.
I20260929 15:48:07.239274 15138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda3cb244e74b4cb65115a60bef039d" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ab64c8a8443088fcd4b91b10a41cd" is_pre_election: true
I20260929 15:48:07.239557 15203 leader_election.cc:302] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b723a663744ca3a5061421f357193e, fa6d7221ff564576a8c02638a00fd681; no voters:
I20260929 15:48:07.239805 15138 raft_consensus.cc:2454] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 0.
I20260929 15:48:07.240051 15337 raft_consensus.cc:2781] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:48:07.240273 15337 raft_consensus.cc:482] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:48:07.240414 15337 raft_consensus.cc:3037] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:07.242884 15337 raft_consensus.cc:504] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.243628 15337 leader_election.cc:288] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 election: Requested vote from peers fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423), 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397)
I20260929 15:48:07.244346 15006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda3cb244e74b4cb65115a60bef039d" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6d7221ff564576a8c02638a00fd681"
I20260929 15:48:07.244491 15138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fda3cb244e74b4cb65115a60bef039d" candidate_uuid: "d6b723a663744ca3a5061421f357193e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822ab64c8a8443088fcd4b91b10a41cd"
I20260929 15:48:07.244577 15006 raft_consensus.cc:3037] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:07.244757 15138 raft_consensus.cc:3037] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:07.247103 15006 raft_consensus.cc:2454] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 1.
I20260929 15:48:07.247382 15138 raft_consensus.cc:2454] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b723a663744ca3a5061421f357193e in term 1.
I20260929 15:48:07.247680 15203 leader_election.cc:302] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b723a663744ca3a5061421f357193e, fa6d7221ff564576a8c02638a00fd681; no voters:
I20260929 15:48:07.248075 15337 raft_consensus.cc:2781] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:07.248577 15337 raft_consensus.cc:686] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 1 LEADER]: Becoming Leader. State: Replica: d6b723a663744ca3a5061421f357193e, State: Running, Role: LEADER
I20260929 15:48:07.249094 15337 consensus_queue.cc:237] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } }
I20260929 15:48:07.254356 14865 catalog_manager.cc:5495] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b723a663744ca3a5061421f357193e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d6b723a663744ca3a5061421f357193e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6b723a663744ca3a5061421f357193e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44519 } health_report { overall_health: HEALTHY } } }
W20260929 15:48:07.281212 15315 tablet.cc:2369] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:48:07.301273 15184 tablet.cc:2369] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:48:07.302104 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.296418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.0.97.3:44519)
0929 15:48:07.296634 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.296643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.296749 (+ 106us) server_negotiation.cc:407] Connection header received
0929 15:48:07.297003 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.297007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.297089 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.297269 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:48:07.297654 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.298371 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.299761 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.300240 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.300371 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.300928 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:48:07.301508 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:48:07.301663 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.301747 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.301837 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:48:07.311473 15006 raft_consensus.cc:1261] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Refusing update from remote peer d6b723a663744ca3a5061421f357193e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:07.311700 15138 raft_consensus.cc:1261] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Refusing update from remote peer d6b723a663744ca3a5061421f357193e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:07.312760 15337 consensus_queue.cc:1035] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [LEADER]: Connected to new peer: Peer: permanent_uuid: "822ab64c8a8443088fcd4b91b10a41cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:07.313148 15340 consensus_queue.cc:1035] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6d7221ff564576a8c02638a00fd681" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:07.326709 15342 mvcc.cc:205] Tried to move back new op lower bound from 7334694450415407104 to 7334694450176610304. Current Snapshot: MvccSnapshot[applied={T|T < 7334694450415407104}]
I20260929 15:48:07.342394 14865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:07.342715 14865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:07.346942 14865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Sending DeleteTablet for 3 replicas of tablet 6fda3cb244e74b4cb65115a60bef039d
I20260929 15:48:07.348461 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fda3cb244e74b4cb65115a60bef039d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60116
I20260929 15:48:07.348461 14986 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fda3cb244e74b4cb65115a60bef039d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55490
I20260929 15:48:07.348798 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14902
I20260929 15:48:07.348773 15249 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fda3cb244e74b4cb65115a60bef039d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41628
I20260929 15:48:07.349819 15352 tablet_replica.cc:330] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: stopping tablet replica
I20260929 15:48:07.350309 15352 raft_consensus.cc:2229] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:07.350864 15352 raft_consensus.cc:2258] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:07.351720 15351 tablet_replica.cc:330] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: stopping tablet replica
I20260929 15:48:07.352145 15351 raft_consensus.cc:2229] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:07.352826 15351 raft_consensus.cc:2258] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:07.353649 15353 tablet_replica.cc:330] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: stopping tablet replica
I20260929 15:48:07.354094 15353 raft_consensus.cc:2229] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:48:07.355057 15353 raft_consensus.cc:2258] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:07.356051 15352 ts_tablet_manager.cc:1905] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:07.357803 15351 ts_tablet_manager.cc:1905] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:07.363866 15352 ts_tablet_manager.cc:1918] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:07.364135 15352 log.cc:1196] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/6fda3cb244e74b4cb65115a60bef039d
I20260929 15:48:07.364811 15352 ts_tablet_manager.cc:1939] T 6fda3cb244e74b4cb65115a60bef039d P 822ab64c8a8443088fcd4b91b10a41cd: Deleting consensus metadata
I20260929 15:48:07.366925 14851 catalog_manager.cc:4841] TS 822ab64c8a8443088fcd4b91b10a41cd (127.0.97.2:37397): tablet 6fda3cb244e74b4cb65115a60bef039d (table test-table [id=f90ef5fc807747e8a05cf9074410f97a]) successfully deleted
I20260929 15:48:07.368301 15353 ts_tablet_manager.cc:1905] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:07.368959 15351 ts_tablet_manager.cc:1918] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:07.369225 15351 log.cc:1196] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/6fda3cb244e74b4cb65115a60bef039d
I20260929 15:48:07.369855 15351 ts_tablet_manager.cc:1939] T 6fda3cb244e74b4cb65115a60bef039d P fa6d7221ff564576a8c02638a00fd681: Deleting consensus metadata
I20260929 15:48:07.371366 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.349740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.0.97.1:35423)
0929 15:48:07.349972 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.349980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.350008 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:48:07.350171 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.350177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.350304 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.350451 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:07.352784 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.354342 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.356573 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.357230 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.357401 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.361613 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:07.361628 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:07.361647 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:07.361714 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:07.366642 (+ 4928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.367663 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.367673 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.367677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.367870 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.368386 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.368393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.368399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.368912 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:07.370539 (+ 1627us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.370708 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.370953 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:48:07.372363 15046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:07.372283 14850 catalog_manager.cc:4841] TS fa6d7221ff564576a8c02638a00fd681 (127.0.97.1:35423): tablet 6fda3cb244e74b4cb65115a60bef039d (table test-table [id=f90ef5fc807747e8a05cf9074410f97a]) successfully deleted
I20260929 15:48:07.377411 15353 ts_tablet_manager.cc:1918] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:07.377678 15353 log.cc:1196] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/6fda3cb244e74b4cb65115a60bef039d
I20260929 15:48:07.378388 15353 ts_tablet_manager.cc:1939] T 6fda3cb244e74b4cb65115a60bef039d P d6b723a663744ca3a5061421f357193e: Deleting consensus metadata
I20260929 15:48:07.381846 14851 catalog_manager.cc:4841] TS d6b723a663744ca3a5061421f357193e (127.0.97.3:44519): tablet 6fda3cb244e74b4cb65115a60bef039d (table test-table [id=f90ef5fc807747e8a05cf9074410f97a]) successfully deleted
I20260929 15:48:07.501588 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14902
I20260929 15:48:07.520792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 15055
I20260929 15:48:07.537053 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.521702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.0.97.2:37397)
0929 15:48:07.521936 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.521944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.521967 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:48:07.522036 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.522041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.522119 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.522234 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:07.523646 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.524574 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.525942 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.526364 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.526511 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.529696 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:07.529713 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:07.529811 (+ 98us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:07.529864 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:07.533693 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.534715 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.534724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.534726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.534879 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.535374 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.535382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.535385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.535805 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:07.536052 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.536483 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.536709 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:48:07.537937 15178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:07.647179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 15055
I20260929 15:48:07.672482 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 15186
I20260929 15:48:07.689946 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:07.673343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.97.3:44519)
0929 15:48:07.673514 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:07.673518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:07.673538 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:48:07.673835 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:07.673840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:07.673913 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:07.674017 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:07.675784 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.676803 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.678769 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:07.679255 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:07.679379 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:07.683076 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:07.683092 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:07.683121 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:07.683171 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:07.686537 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.687646 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.687653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.687655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.687850 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:07.688324 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:07.688328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:07.688330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:07.688699 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:07.688932 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:07.689318 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 15:48:07.689508 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:48:07.690739 15309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:07.821201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 15186
I20260929 15:48:07.841950 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 14832
I20260929 15:48:07.843058 14895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:07.961632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 14832
I20260929 15:48:07.985131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39295
--webserver_interface=127.0.97.62
--webserver_port=34479
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:08.342522 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:08.342849 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:08.343016 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:08.351558 15358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:48:08.351714 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:08.351769 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:08.351876 15358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:48:08.351958 15358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:48:08.363212 15358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:08.364769 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:08.366221 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:08.379571 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:08.379793 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:08.379572 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:08.381037 15358 server_base.cc:1034] running on GCE node
I20260929 15:48:08.381989 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:08.383576 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:08.384801 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790696888384743 us; error 51 us; skew 500 ppm
I20260929 15:48:08.385147 15358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:08 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:08.388398 15358 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:48:08.390007 15358 webserver.cc:460] Webserver started at http://127.0.97.62:34479/ using document root <none> and password file <none>
I20260929 15:48:08.390748 15358 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:08.390890 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:08.397322 15358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:48:08.400475 15373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:08.401964 15358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:48:08.402204 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "e5b3b5a7abd64ff09a438d260b2bea52"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "f248af9947876c958548590989b6a55e"
server_key_iv: "cbb4ad8f20f40991f4be0d91e081d3a1"
server_key_version: "encryptionkey@0"
I20260929 15:48:08.403482 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:08.425971 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:08.430236 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:08.430555 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:08.450695 15358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39295
I20260929 15:48:08.450730 15424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39295 every 8 connection(s)
I20260929 15:48:08.452406 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:48:08.459487 15425 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:48:08.461532 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15358
I20260929 15:48:08.462348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35423
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39295
--webserver_port=39675
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:48:08.463073 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap starting.
I20260929 15:48:08.484020 15425 log.cc:824] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:08.500960 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:48:08.501560 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52: Bootstrap complete.
I20260929 15:48:08.511241 15425 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:08.512292 15425 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b3b5a7abd64ff09a438d260b2bea52, State: Initialized, Role: FOLLOWER
I20260929 15:48:08.513003 15425 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:08.513324 15425 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:48:08.513478 15425 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:48:08.513674 15425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:48:08.517889 15425 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:08.518428 15425 leader_election.cc:302] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b3b5a7abd64ff09a438d260b2bea52; no voters:
I20260929 15:48:08.519119 15425 leader_election.cc:288] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 15:48:08.519265 15429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 15:48:08.521855 15425 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:48:08.526537 15429 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [term 3 LEADER]: Becoming Leader. State: Replica: e5b3b5a7abd64ff09a438d260b2bea52, State: Running, Role: LEADER
I20260929 15:48:08.527321 15429 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } }
I20260929 15:48:08.530619 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } } }
I20260929 15:48:08.531109 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:08.531831 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b3b5a7abd64ff09a438d260b2bea52. Latest consensus state: current_term: 3 leader_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b3b5a7abd64ff09a438d260b2bea52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39295 } } }
I20260929 15:48:08.532217 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b3b5a7abd64ff09a438d260b2bea52 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:08.535971 15440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:48:08.545264 15440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:48:08.547286 15440 catalog_manager.cc:1259] Loaded cluster ID: a7dd12db8e834adcbebd91064573469f
I20260929 15:48:08.547425 15440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 15:48:08.550210 15440 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f32836379ad google::LogMessage::Fail() at ??:0
@ 0x7f328363a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f32836374de google::LogMessage::Flush() at ??:0
@ 0x7f3283638a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f329906a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3299067976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f32990ef610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f32990ef439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3286283345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f328628691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3286286799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3286261eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f328953b6db start_thread at ??:0
@ 0x7f32824d161f clone at ??:0
W20260929 15:48:08.871881 15427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:08.872308 15427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:08.872488 15427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:08.880247 15427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:08.880427 15427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:08.880620 15427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:48:08.890965 15427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39675
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:08.892448 15427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:08.893846 15427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:08.908454 15467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:08.908542 15468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:08.908370 15470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:08.909550 15427 server_base.cc:1034] running on GCE node
I20260929 15:48:08.910200 15427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:08.911216 15427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:08.912434 15427 hybrid_clock.cc:648] HybridClock initialized: now 1790696888912382 us; error 35 us; skew 500 ppm
I20260929 15:48:08.912806 15427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:08 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:08.923280 15427 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:48:08.928689 15427 webserver.cc:460] Webserver started at http://127.0.97.1:39675/ using document root <none> and password file <none>
I20260929 15:48:08.929595 15427 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:08.929786 15427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:08.937214 15427 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:48:08.941516 15477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:08.943457 15427 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:48:08.943670 15427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "fa6d7221ff564576a8c02638a00fd681"
format_stamp: "Formatted at 2026-09-29 15:48:02 on dist-test-slave-7491"
server_key: "0f53433f2448a81bd549d9fdc73224e6"
server_key_iv: "bcb94edcbc169abc6113cc8ba09bc92d"
server_key_version: "encryptionkey@0"
I20260929 15:48:08.945312 15427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:08.979952 15427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:08.984128 15427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:08.984459 15427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:08.985435 15427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:08.986770 15427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:08.986868 15427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:08.986948 15427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:08.987003 15427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.022315 15427 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35423
I20260929 15:48:09.022415 15589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35423 every 8 connection(s)
I20260929 15:48:09.023862 15427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:48:09.028596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15427
I20260929 15:48:09.029382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37397
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39295
--webserver_port=38269
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:48:09.029661 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.027198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:41751)
0929 15:48:09.028079 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.028405 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":316,"threads_started":1}
W20260929 15:48:09.030460 15590 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39295 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39295: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
I20260929 15:48:09.031538 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.031067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:60485)
0929 15:48:09.031277 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.031375 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 15:48:09.032688 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.032243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:49797)
0929 15:48:09.032415 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.032521 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:48:09.033017 15590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:48:09.388510 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:09.388819 15593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:09.388990 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:09.396771 15593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:09.396889 15593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:09.397080 15593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:48:09.407204 15593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38269
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:09.408664 15593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:09.409997 15593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:09.424082 15598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:09.424225 15599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:09.424139 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:09.424925 15593 server_base.cc:1034] running on GCE node
I20260929 15:48:09.425546 15593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:09.426563 15593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:09.427788 15593 hybrid_clock.cc:648] HybridClock initialized: now 1790696889427746 us; error 37 us; skew 500 ppm
I20260929 15:48:09.428117 15593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:09 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:09.431658 15593 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:48:09.433384 15593 webserver.cc:460] Webserver started at http://127.0.97.2:38269/ using document root <none> and password file <none>
I20260929 15:48:09.434144 15593 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:09.434289 15593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:09.439998 15593 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:48:09.442946 15608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.444651 15593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:48:09.444857 15593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "822ab64c8a8443088fcd4b91b10a41cd"
format_stamp: "Formatted at 2026-09-29 15:48:03 on dist-test-slave-7491"
server_key: "df7540d6020b2130557b4d92fa27a8c6"
server_key_iv: "6872f9db4895c2692bdd225313c21945"
server_key_version: "encryptionkey@0"
I20260929 15:48:09.445936 15593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:09.461385 15593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:09.465473 15593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:09.465767 15593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:09.466796 15593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:09.468221 15593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:09.468335 15593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.468468 15593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:09.468562 15593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.503640 15593 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37397
I20260929 15:48:09.503849 15720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37397 every 8 connection(s)
I20260929 15:48:09.505199 15593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:48:09.507519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15593
I20260929 15:48:09.508340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44519
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39295
--webserver_port=45379
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:48:09.510852 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.508790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:37631)
0929 15:48:09.509637 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.509904 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":362,"threads_started":1}
W20260929 15:48:09.511515 15721 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39295 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39295: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
I20260929 15:48:09.512537 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.512193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:44905)
0929 15:48:09.512351 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.512423 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 15:48:09.513487 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.513167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:60503)
0929 15:48:09.513296 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.513370 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:48:09.513789 15721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:48:09.866467 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:09.866806 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:09.866911 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:09.874845 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:09.874992 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:09.875113 15724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:48:09.885377 15724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45379
--tserver_master_addrs=127.0.97.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:09.886729 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:09.888033 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:09.902395 15729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:09.902482 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:09.902455 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:09.903920 15724 server_base.cc:1034] running on GCE node
I20260929 15:48:09.904670 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:09.905710 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:09.906980 15724 hybrid_clock.cc:648] HybridClock initialized: now 1790696889906930 us; error 42 us; skew 500 ppm
I20260929 15:48:09.907373 15724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:09 dist-test-slave-7491 krb5kdc[14280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:09.911018 15724 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:48:09.912904 15724 webserver.cc:460] Webserver started at http://127.0.97.3:45379/ using document root <none> and password file <none>
I20260929 15:48:09.913667 15724 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:09.913829 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:09.919569 15724 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:48:09.923091 15739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.924991 15724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:48:09.925179 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "d6b723a663744ca3a5061421f357193e"
format_stamp: "Formatted at 2026-09-29 15:48:04 on dist-test-slave-7491"
server_key: "047115d564e07dd9914e4be5ad0f670e"
server_key_iv: "5aa3cc6cc9c5e1c501d1d992e536a8ea"
server_key_version: "encryptionkey@0"
I20260929 15:48:09.926214 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:09.942296 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:09.947781 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:09.948068 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:09.949206 15724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:09.950829 15724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:09.950924 15724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.951048 15724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:09.951221 15724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:09.988519 15724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44519
I20260929 15:48:09.988603 15851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44519 every 8 connection(s)
I20260929 15:48:09.990127 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:48:09.995460 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.993286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:34987)
0929 15:48:09.994103 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.994314 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":342,"threads_started":1}
I20260929 15:48:09.995501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15724
W20260929 15:48:09.996356 15852 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39295 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39295: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
I20260929 15:48:09.997282 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.996906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:33839)
0929 15:48:09.997076 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.997159 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 15:48:09.998445 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:09.997894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:53259)
0929 15:48:09.998217 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:09.998320 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 15:48:09.998762 15852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 15:48:10.034519 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:10.033616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:58597)
0929 15:48:10.034141 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:10.034272 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":266,"threads_started":1}
I20260929 15:48:10.515298 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:10.514390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:49649)
0929 15:48:10.514921 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:10.515018 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":279,"threads_started":1}
I20260929 15:48:11.000382 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:10.999414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:39505)
0929 15:48:10.999931 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:11.000028 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
I20260929 15:48:11.037029 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:11.035931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:60461)
0929 15:48:11.036769 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:11.036851 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":581,"threads_started":1}
I20260929 15:48:11.517138 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:11.516393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:42663)
0929 15:48:11.516868 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:11.516960 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
I20260929 15:48:12.002370 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:12.001487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:35069)
0929 15:48:12.002010 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:12.002111 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
I20260929 15:48:12.038708 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:12.038007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:51249)
0929 15:48:12.038441 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:12.038523 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":236,"threads_started":1}
I20260929 15:48:12.518882 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:12.518121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:56795)
0929 15:48:12.518615 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:12.518701 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260929 15:48:13.004660 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:13.003520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:47675)
0929 15:48:13.004263 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:13.004386 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":433,"threads_started":1}
I20260929 15:48:13.040872 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:13.039887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:35819)
0929 15:48:13.040481 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:13.040620 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":344,"threads_started":1}
I20260929 15:48:13.520862 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:13.519945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:60523)
0929 15:48:13.520559 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:13.520658 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":314,"threads_started":1}
I20260929 15:48:14.006726 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:14.005960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:50287)
0929 15:48:14.006464 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:14.006546 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260929 15:48:14.042757 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:14.041886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:56927)
0929 15:48:14.042389 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:14.042478 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260929 15:48:14.522799 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:14.521865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:43833)
0929 15:48:14.522530 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:14.522617 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":259,"threads_started":1}
I20260929 15:48:15.008901 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:15.007976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:39877)
0929 15:48:15.008586 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:15.008710 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":377,"threads_started":1}
I20260929 15:48:15.044979 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:15.044037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:50485)
0929 15:48:15.044649 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:15.044780 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":388,"threads_started":1}
I20260929 15:48:15.525099 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:15.523942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:44731)
0929 15:48:15.524638 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:15.524880 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":393,"threads_started":1}
I20260929 15:48:16.010859 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:16.009996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:45215)
0929 15:48:16.010547 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:16.010660 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260929 15:48:16.047065 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:16.046165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:42031)
0929 15:48:16.046752 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:16.046850 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":262,"threads_started":1}
I20260929 15:48:16.527083 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:16.526188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:59907)
0929 15:48:16.526737 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:16.526832 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":240,"threads_started":1}
I20260929 15:48:17.012928 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:17.012005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:52077)
0929 15:48:17.012592 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:17.012706 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260929 15:48:17.049558 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:17.048425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:55193)
0929 15:48:17.049208 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:17.049372 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":549,"threads_started":1}
I20260929 15:48:17.529002 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:17.528253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:53013)
0929 15:48:17.528747 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:17.528833 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":223,"threads_started":1}
I20260929 15:48:18.014671 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:18.013874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:60551)
0929 15:48:18.014409 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:18.014487 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":301,"threads_started":1}
I20260929 15:48:18.051848 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:18.050896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:55649)
0929 15:48:18.051531 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:18.051618 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":388,"threads_started":1}
I20260929 15:48:18.531186 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:18.530156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:54371)
0929 15:48:18.530787 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:18.530899 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":321,"threads_started":1}
I20260929 15:48:19.016678 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:19.015645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:54993)
0929 15:48:19.016275 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:19.016396 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":285,"threads_started":1}
I20260929 15:48:19.053646 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:19.052946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:53787)
0929 15:48:19.053403 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:19.053483 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":261,"threads_started":1}
I20260929 15:48:19.532860 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:19.532105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:39233)
0929 15:48:19.532598 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:19.532695 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":289,"threads_started":1}
I20260929 15:48:20.018545 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:20.017781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:36107)
0929 15:48:20.018282 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:20.018372 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
I20260929 15:48:20.055867 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:20.054855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:58199)
0929 15:48:20.055538 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:20.055661 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":377,"threads_started":1}
I20260929 15:48:20.534490 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:20.533755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:35657)
0929 15:48:20.534227 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:20.534310 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260929 15:48:21.020641 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:21.019734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:55587)
0929 15:48:21.020311 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:21.020454 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260929 15:48:21.057610 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:21.056939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:33659)
0929 15:48:21.057367 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:21.057445 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260929 15:48:21.536445 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:21.535584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:34999)
0929 15:48:21.536126 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:21.536272 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":322,"threads_started":1}
I20260929 15:48:22.022639 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:22.021883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:52621)
0929 15:48:22.022370 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:22.022464 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260929 15:48:22.059653 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:22.058782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:44393)
0929 15:48:22.059361 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:22.059453 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":312,"threads_started":1}
I20260929 15:48:22.538198 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:22.537386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:34773)
0929 15:48:22.537933 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:22.538015 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":300,"threads_started":1}
I20260929 15:48:23.024596 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:23.023717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:47263)
0929 15:48:23.024320 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:23.024410 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
I20260929 15:48:23.061308 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:23.060575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:46435)
0929 15:48:23.061029 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:23.061111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260929 15:48:23.539945 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:23.539217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:47333)
0929 15:48:23.539649 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:23.539744 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260929 15:48:24.026295 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:24.025476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:47501)
0929 15:48:24.025973 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:24.026072 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260929 15:48:24.063421 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:24.062591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.1:50599)
0929 15:48:24.063128 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:24.063228 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":290,"threads_started":1}
I20260929 15:48:24.541950 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:24.541172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.2:59619)
0929 15:48:24.541650 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:24.541764 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260929 15:48:25.005586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 15427
I20260929 15:48:25.028590 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:25.027565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39295 (local address 127.0.97.3:40773)
0929 15:48:25.028224 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:25.028348 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":339,"threads_started":1}
I20260929 15:48:25.034765 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:25.007542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.0.97.1:35423)
0929 15:48:25.008257 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:25.008278 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:25.008330 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:48:25.010508 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:25.010520 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:25.011193 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:25.011547 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:25.013233 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:25.014551 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:25.015953 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:25.016795 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:25.016954 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:25.026468 (+ 9514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:25.026481 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:25.026504 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:25.026568 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:25.030717 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:25.031632 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:25.031641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:25.031649 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:25.031946 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:25.032602 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:25.032607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:25.032609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:25.033211 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:25.033469 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:25.034085 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 15:48:25.034459 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":328,"threads_started":1}
I20260929 15:48:25.036414 15585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:25.127038 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 15427
I20260929 15:48:25.145239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 15593
I20260929 15:48:25.165650 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:25.146394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.97.2:37397)
0929 15:48:25.146933 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:25.146947 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:25.146985 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:48:25.148723 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:25.148735 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:25.149189 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:25.149511 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:25.151169 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:25.152675 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:25.153992 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:25.154699 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:25.154854 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:25.158264 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:25.158273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:25.158292 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:25.158340 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:25.161895 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:25.162783 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:25.162791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:25.162798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:25.162991 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:25.163442 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:25.163446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:25.163448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:25.164073 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:25.164418 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:25.164885 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 15:48:25.165359 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":265,"threads_started":1}
I20260929 15:48:25.167166 15716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:25.253643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 15593
I20260929 15:48:25.272292 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 15724
I20260929 15:48:25.291800 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:25.273456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.97.3:44519)
0929 15:48:25.274085 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:25.274099 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:25.274157 (+ 58us) server_negotiation.cc:407] Connection header received
0929 15:48:25.276000 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:25.276012 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:25.276599 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:25.276999 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:25.278505 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:25.279778 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:25.281150 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:25.281870 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:25.282106 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:25.285071 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:25.285080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:25.285097 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:25.285142 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:25.288127 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:25.289004 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:25.289014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:25.289024 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:25.289218 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:25.289630 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:25.289636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:25.289640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:25.290210 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:25.290540 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:25.291023 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 15:48:25.291468 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":318,"threads_started":1}
I20260929 15:48:25.293449 15847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:25.381747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 15724
2026-09-29T15:48:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24875 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:48:25 dist-test-slave-7491 krb5kdc[15910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:48:25 dist-test-slave-7491 krb5kdc[15910](info): set up 2 sockets
Sep 29 15:48:25 dist-test-slave-7491 krb5kdc[15910](info): commencing operation
krb5kdc: starting...
W20260929 15:48:26.532832 388 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:48:26 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:48:26.785480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:27.139879 15923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:27.140355 15923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:27.140532 15923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:27.148649 15923 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 15:48:27.148780 15923 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 15:48:27.148831 15923 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 15:48:27.148943 15923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:48:27.149050 15923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:27.149168 15923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:27.149248 15923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:48:27.149333 15923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:48:27.159935 15923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:27.161434 15923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:27.162818 15923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:27.176565 15928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:27.176647 15929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:27.176592 15931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:27.176935 15923 server_base.cc:1034] running on GCE node
I20260929 15:48:27.177924 15923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:27.179605 15923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:27.180949 15923 hybrid_clock.cc:648] HybridClock initialized: now 1790696907180870 us; error 42 us; skew 500 ppm
I20260929 15:48:27.181368 15923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:27 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:27.184901 15923 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:48:27.186510 15923 webserver.cc:460] Webserver started at http://127.0.97.62:44363/ using document root <none> and password file <none>
I20260929 15:48:27.187305 15923 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:27.187451 15923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:27.187788 15923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:27.191131 15923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "4f05b0d205ac435d97f846775b5dd097"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "baf9c738deaf607d2e3c759692110d0b"
server_key_iv: "9c4db492dc4a1ca84d949edd910d1a18"
server_key_version: "encryptionkey@0"
I20260929 15:48:27.192013 15923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f05b0d205ac435d97f846775b5dd097"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "baf9c738deaf607d2e3c759692110d0b"
server_key_iv: "9c4db492dc4a1ca84d949edd910d1a18"
server_key_version: "encryptionkey@0"
I20260929 15:48:27.197817 15923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:48:27.202417 15938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:27.204139 15923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 15:48:27.204487 15923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "4f05b0d205ac435d97f846775b5dd097"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "baf9c738deaf607d2e3c759692110d0b"
server_key_iv: "9c4db492dc4a1ca84d949edd910d1a18"
server_key_version: "encryptionkey@0"
I20260929 15:48:27.204872 15923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:27.222874 15923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:27.227870 15923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:27.228299 15923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:27.249202 15989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45651 every 8 connection(s)
I20260929 15:48:27.249195 15923 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45651
I20260929 15:48:27.250911 15923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:48:27.252765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 15923
I20260929 15:48:27.253005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:48:27.253628 388 external_mini_cluster.cc:1428] Setting key 90d3ed12f4854a5704165fbcb83b2721
I20260929 15:48:27.256027 15990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:48:27 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:27.272542 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap starting.
I20260929 15:48:27.277428 15990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:27.278816 15990 log.cc:824] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:27.282639 15990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: No bootstrap required, opened a new log
I20260929 15:48:27.283103 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:27.255921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.0.97.62:45651)
0929 15:48:27.256988 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:27.257005 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:27.257068 (+ 63us) server_negotiation.cc:407] Connection header received
0929 15:48:27.259624 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:27.259660 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:27.260333 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:27.260882 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:27.262432 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:27.264963 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:27.266684 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:27.267939 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:27.268230 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:27.272978 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:27.272990 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:27.273011 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:27.273066 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:27.277428 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:27.278374 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:27.278383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:27.278393 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:27.278663 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:27.279266 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:27.279270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:27.279273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:27.280006 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:27.280371 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:27.280884 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 15:48:27.281585 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":506,"threads_started":1}
I20260929 15:48:27.292783 15990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:27.293344 15990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:27.293464 15990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f05b0d205ac435d97f846775b5dd097, State: Initialized, Role: FOLLOWER
I20260929 15:48:27.294163 15990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:27.294399 15990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:48:27.294495 15990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:48:27.294631 15990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:27.297158 15990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:27.297840 15990 leader_election.cc:302] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f05b0d205ac435d97f846775b5dd097; no voters:
I20260929 15:48:27.298566 15990 leader_election.cc:288] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:48:27.298768 15995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:27.299495 15995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 LEADER]: Becoming Leader. State: Replica: 4f05b0d205ac435d97f846775b5dd097, State: Running, Role: LEADER
I20260929 15:48:27.300107 15995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:27.300976 15990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:48:27.303303 15997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f05b0d205ac435d97f846775b5dd097. Latest consensus state: current_term: 1 leader_uuid: "4f05b0d205ac435d97f846775b5dd097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260929 15:48:27.303292 15996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f05b0d205ac435d97f846775b5dd097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260929 15:48:27.304071 15997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:27.304067 15996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:27.305114 16002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:48:27.312441 16002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:48:27.325054 16002 catalog_manager.cc:1347] Generated new cluster ID: 62cf2d7d77b94e7b9825e65bee0dece4
I20260929 15:48:27.325193 16002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:48:27.337298 16002 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:48:27.339322 16002 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:48:27.354022 16002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Generated new TSK 0
I20260929 15:48:27.355223 16002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:48:27.422034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:27.832324 16016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:27.832716 16016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:27.832918 16016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:27.840911 16016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:27.841101 16016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:27.841310 16016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:48:27.853411 16016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:27.854923 16016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:27.856319 16016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:27.870828 16021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:27.870904 16024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:27.871385 16022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:27.871631 16016 server_base.cc:1034] running on GCE node
I20260929 15:48:27.873253 16016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:27.874522 16016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:27.875874 16016 hybrid_clock.cc:648] HybridClock initialized: now 1790696907875816 us; error 75 us; skew 500 ppm
I20260929 15:48:27.876389 16016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:27 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:27.880308 16016 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:48:27.882105 16016 webserver.cc:460] Webserver started at http://127.0.97.1:35217/ using document root <none> and password file <none>
I20260929 15:48:27.882900 16016 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:27.883066 16016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:27.883410 16016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:27.887283 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "0a6bbc005d6a468d9aed346a14b292fa"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "69d31b414367b4a066c95b6b8476dac6"
server_key_iv: "21027ef3f12d58d33cfd7c53249467f0"
server_key_version: "encryptionkey@0"
I20260929 15:48:27.888628 16016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0a6bbc005d6a468d9aed346a14b292fa"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "69d31b414367b4a066c95b6b8476dac6"
server_key_iv: "21027ef3f12d58d33cfd7c53249467f0"
server_key_version: "encryptionkey@0"
I20260929 15:48:27.896034 16016 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 15:48:27.900672 16031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:27.902966 16016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 15:48:27.903326 16016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "0a6bbc005d6a468d9aed346a14b292fa"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "69d31b414367b4a066c95b6b8476dac6"
server_key_iv: "21027ef3f12d58d33cfd7c53249467f0"
server_key_version: "encryptionkey@0"
I20260929 15:48:27.903663 16016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:27.930050 16016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:27.936131 16016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:27.936561 16016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:27.937773 16016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:27.939437 16016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:27.939568 16016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:27.939745 16016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:27.939826 16016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:27.982657 16016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34787
I20260929 15:48:27.982676 16143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34787 every 8 connection(s)
I20260929 15:48:27.984046 16016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:48:27.994032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16016
I20260929 15:48:27.994288 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:48:27.995100 388 external_mini_cluster.cc:1428] Setting key 43f9316b694d9e8a4ce37141ae5cf0ec
Sep 29 15:48:27 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:28.012907 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:27.987458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:49355)
0929 15:48:27.988430 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:27.988479 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:27.990257 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:27.991141 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:27.991151 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:27.991799 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:27.992482 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:27.992502 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:27.993832 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:27.993839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:27.995993 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:27.996011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:27.996993 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:27.997002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:27.997304 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:27.998173 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:27.998221 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:28.002156 (+ 3935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:28.007236 (+ 5080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:28.007249 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:28.007278 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:28.008041 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:28.008654 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:28.008658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:28.008661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:28.008955 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:28.009813 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:28.009824 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:28.010375 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 15:48:28.010905 (+ 530us) client_negotiation.cc:240] Negotiation successful
0929 15:48:28.011518 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":440,"threads_started":1}
I20260929 15:48:28.013145 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:27.987103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49355 (local address 127.0.97.62:45651)
0929 15:48:27.987824 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:27.987857 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:27.988609 (+ 752us) server_negotiation.cc:407] Connection header received
0929 15:48:27.990618 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:27.990623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:27.990716 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:27.990828 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:27.992787 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:27.993597 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:27.996227 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:27.996693 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:27.996890 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:28.002594 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:28.002603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:28.002608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:28.002667 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:28.006918 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:28.008289 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:28.008299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:28.008307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:28.008486 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:28.009171 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:28.009178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:28.009184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:28.009601 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:28.009770 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:28.012459 (+ 2689us) server_negotiation.cc:299] Negotiation successful
0929 15:48:28.012736 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":512,"thread_start_us":310,"threads_started":1}
I20260929 15:48:28.015117 16144 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260929 15:48:28.015479 16144 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:28.016381 16144 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260929 15:48:28.019189 15955 ts_manager.cc:194] Registered new tserver with Master: 0a6bbc005d6a468d9aed346a14b292fa (127.0.97.1:34787)
I20260929 15:48:28.021620 15955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49355
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:48:28.056030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:28.415257 16150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:28.415578 16150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:28.415675 16150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:28.423448 16150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:28.423586 16150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:28.423703 16150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:48:28.433984 16150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:28.435300 16150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:28.436961 16150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:28.452925 16155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:28.452869 16156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:28.452965 16158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:28.453871 16150 server_base.cc:1034] running on GCE node
I20260929 15:48:28.454560 16150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:28.455602 16150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:28.456893 16150 hybrid_clock.cc:648] HybridClock initialized: now 1790696908456818 us; error 57 us; skew 500 ppm
I20260929 15:48:28.457283 16150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:28 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:28.461045 16150 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:48:28.462932 16150 webserver.cc:460] Webserver started at http://127.0.97.2:46473/ using document root <none> and password file <none>
I20260929 15:48:28.463696 16150 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:28.463851 16150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:28.464282 16150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:28.467569 16150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "5038ba53e7f04ca782d2d05afb291742"
format_stamp: "Formatted at 2026-09-29 15:48:28 on dist-test-slave-7491"
server_key: "2313bcb4c461655ac7cc2929dce83741"
server_key_iv: "4c8559342322a7680480943c51baf32e"
server_key_version: "encryptionkey@0"
I20260929 15:48:28.468556 16150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5038ba53e7f04ca782d2d05afb291742"
format_stamp: "Formatted at 2026-09-29 15:48:28 on dist-test-slave-7491"
server_key: "2313bcb4c461655ac7cc2929dce83741"
server_key_iv: "4c8559342322a7680480943c51baf32e"
server_key_version: "encryptionkey@0"
I20260929 15:48:28.474663 16150 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:48:28.478811 16165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:28.480546 16150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:48:28.480782 16150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "5038ba53e7f04ca782d2d05afb291742"
format_stamp: "Formatted at 2026-09-29 15:48:28 on dist-test-slave-7491"
server_key: "2313bcb4c461655ac7cc2929dce83741"
server_key_iv: "4c8559342322a7680480943c51baf32e"
server_key_version: "encryptionkey@0"
I20260929 15:48:28.481010 16150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:28.505276 16150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:28.512938 16150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:28.513309 16150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:28.514552 16150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:28.516016 16150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:28.516108 16150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:28.516278 16150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:28.516337 16150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:28.554191 16150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42089
I20260929 15:48:28.554246 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42089 every 8 connection(s)
I20260929 15:48:28.555619 16150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:48:28.563596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16150
I20260929 15:48:28.563884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:48:28.564431 388 external_mini_cluster.cc:1428] Setting key 0939969eee4b4f70ede60303f6c21d6b
Sep 29 15:48:28 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:28.578145 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:28.558529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40819 (local address 127.0.97.62:45651)
0929 15:48:28.559005 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:28.559009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:28.559854 (+ 845us) server_negotiation.cc:407] Connection header received
0929 15:48:28.561901 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:28.561908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:28.562012 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:28.562146 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:28.563812 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:28.564916 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:28.566645 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:28.567087 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:28.567240 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:28.571327 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:28.571333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:28.571336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:28.571376 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:28.574662 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:28.575579 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:28.575586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:28.575589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:28.575736 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:28.576266 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:28.576271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:28.576275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:28.576570 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:28.576758 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:28.577514 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 15:48:28.577767 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260929 15:48:28.579208 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:28.558909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:40819)
0929 15:48:28.559658 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:28.559708 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:28.561554 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:28.562408 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:28.562417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:28.562997 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:28.563599 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:28.563622 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:28.565110 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:28.565121 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:28.566458 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:28.566474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:28.567379 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:28.567389 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:28.567642 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:28.568480 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:28.568506 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:28.570989 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:28.574876 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:28.574887 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:28.574914 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:28.575415 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:28.575869 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:28.575874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:28.575878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:28.576096 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:28.576726 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:28.576735 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:28.577117 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 15:48:28.577410 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 15:48:28.577995 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":311,"threads_started":1}
I20260929 15:48:28.581139 16278 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260929 15:48:28.581522 16278 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:28.582346 16278 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260929 15:48:28.584992 15955 ts_manager.cc:194] Registered new tserver with Master: 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
I20260929 15:48:28.586514 15955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40819
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:48:28.624366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:28.976766 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:28.977082 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:28.977178 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:28.987980 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:28.988131 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:28.988356 16284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:48:28.999955 16284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:29.001466 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:29.002916 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:29.018211 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:29.018278 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:29.018239 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:29.019315 16284 server_base.cc:1034] running on GCE node
I20260929 15:48:29.020112 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:29.021313 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:29.022571 16284 hybrid_clock.cc:648] HybridClock initialized: now 1790696909022526 us; error 45 us; skew 500 ppm
I20260929 15:48:29.022980 16284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:48:29.025689 16144 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
Sep 29 15:48:29 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:29.028147 16284 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:48:29.033696 16284 webserver.cc:460] Webserver started at http://127.0.97.3:36299/ using document root <none> and password file <none>
I20260929 15:48:29.034894 16284 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:29.035074 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:29.035492 16284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:48:29.040318 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "ff7743f2466e4cb88c849b6d7af1230b"
format_stamp: "Formatted at 2026-09-29 15:48:29 on dist-test-slave-7491"
server_key: "8a527628e3eaf7734994a583125de03a"
server_key_iv: "a59ceb614e0741fb63fa3b559d116894"
server_key_version: "encryptionkey@0"
I20260929 15:48:29.041646 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ff7743f2466e4cb88c849b6d7af1230b"
format_stamp: "Formatted at 2026-09-29 15:48:29 on dist-test-slave-7491"
server_key: "8a527628e3eaf7734994a583125de03a"
server_key_iv: "a59ceb614e0741fb63fa3b559d116894"
server_key_version: "encryptionkey@0"
I20260929 15:48:29.049945 16284 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260929 15:48:29.054905 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:29.057065 16284 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:48:29.057322 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "ff7743f2466e4cb88c849b6d7af1230b"
format_stamp: "Formatted at 2026-09-29 15:48:29 on dist-test-slave-7491"
server_key: "8a527628e3eaf7734994a583125de03a"
server_key_iv: "a59ceb614e0741fb63fa3b559d116894"
server_key_version: "encryptionkey@0"
I20260929 15:48:29.057610 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:29.082739 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:29.088315 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:29.088646 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:29.089893 16284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:29.091689 16284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:29.091782 16284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:29.091950 16284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:29.092018 16284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:29.132936 16284 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34941
I20260929 15:48:29.132977 16411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34941 every 8 connection(s)
I20260929 15:48:29.134326 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:48:29.142688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16284
I20260929 15:48:29.142915 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:48:29.143488 388 external_mini_cluster.cc:1428] Setting key a0785c02c9c0dd5963be8fa93877ca10
Sep 29 15:48:29 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:29.161952 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.137623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46851 (local address 127.0.97.62:45651)
0929 15:48:29.138135 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.138141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.139282 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 15:48:29.141440 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.141446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.141558 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.141719 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:29.144321 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.145317 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.149130 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.149592 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.149892 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.154670 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:29.154676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:29.154680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:29.154725 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:29.157611 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.158628 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.158633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.158636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.158827 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.159602 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.159619 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.159623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.160016 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:29.160241 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.161279 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.161551 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":228,"threads_started":1}
I20260929 15:48:29.162917 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.138238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:46851)
0929 15:48:29.139069 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.139129 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:29.141003 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:29.142035 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:29.142045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:29.142988 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:29.143960 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.143994 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.145734 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.145741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.148752 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.148786 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.150026 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.150033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.150319 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.151270 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:29.151299 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:29.154350 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:29.157853 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:29.157862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:29.157878 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:29.158414 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:29.159043 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:29.159053 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:29.159057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:29.159369 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:29.160222 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:29.160231 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:29.160730 (+ 499us) client_negotiation.cc:769] Sending connection context
0929 15:48:29.161136 (+ 406us) client_negotiation.cc:240] Negotiation successful
0929 15:48:29.161735 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":357,"threads_started":1}
I20260929 15:48:29.164779 16412 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260929 15:48:29.165299 16412 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:29.166136 16412 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260929 15:48:29.168269 15955 ts_manager.cc:194] Registered new tserver with Master: ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941)
I20260929 15:48:29.169171 15955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46851
I20260929 15:48:29.175433 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:48:29.196625 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.181026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.0.97.62:45651)
0929 15:48:29.181328 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.181337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.181660 (+ 323us) server_negotiation.cc:407] Connection header received
0929 15:48:29.181897 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.181902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.181973 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.182093 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:29.183561 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.184850 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.186552 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.186976 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.187149 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.190147 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:29.190152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:29.190156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:29.190196 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:29.193258 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.194142 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.194148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.194151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.194297 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.194723 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.194730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.194733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.195093 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:29.195619 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.195963 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.196292 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 15:48:29.201771 15955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:48:29.209699 15955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:48:29.239308 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.226293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.0.97.3:34941)
0929 15:48:29.229197 (+ 2904us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.229205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.229232 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:48:29.229348 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.229357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.229632 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.229835 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:29.231161 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.232861 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.236347 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.238006 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.238139 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.238398 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.238714 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.238924 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2715,"thread_start_us":5085,"threads_started":1}
I20260929 15:48:29.239403 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.226145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34941 (local address 127.0.0.1:48960)
0929 15:48:29.227083 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.227141 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:29.227377 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:29.230161 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:29.230170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:29.230222 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:29.230915 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.230930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.233047 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.233053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.236104 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.236118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.238232 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.238241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.238454 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.238513 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 15:48:29.238662 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 15:48:29.238903 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":301,"spinlock_wait_cycles":2816,"thread_start_us":805,"threads_started":1}
I20260929 15:48:29.240065 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.229675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.97.2:42089)
0929 15:48:29.230820 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.230831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.231590 (+ 759us) server_negotiation.cc:407] Connection header received
0929 15:48:29.231837 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.231844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.232134 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.232405 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:29.233150 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.234627 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.237267 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.238339 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.238470 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.238650 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.239516 (+ 866us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.239714 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":1363,"threads_started":1}
I20260929 15:48:29.240777 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.226825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41106)
0929 15:48:29.229034 (+ 2209us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.229055 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:29.229276 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:29.230161 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:29.230170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:29.230224 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:29.231757 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.231771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.233592 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.233597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.236359 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.236375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.239942 (+ 3567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.239947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.240148 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.240222 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 15:48:29.240315 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:48:29.240432 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"mutex_wait_us":189,"spinlock_wait_cycles":9984,"thread_start_us":1903,"threads_started":1}
I20260929 15:48:29.241319 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.229147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42089 (local address 127.0.0.1:38444)
0929 15:48:29.231461 (+ 2314us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.231481 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:29.231622 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:29.232546 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:29.232551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:29.232578 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:29.232959 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.232974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.234835 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.234841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.237075 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.237089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.238795 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.238801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.238968 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.239000 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:48:29.239098 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:48:29.239233 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2185,"mutex_wait_us":84,"thread_start_us":2046,"threads_started":1}
I20260929 15:48:29.242645 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.227094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.0.97.1:34787)
0929 15:48:29.228660 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.228673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.229155 (+ 482us) server_negotiation.cc:407] Connection header received
0929 15:48:29.229461 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.229467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.229640 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.229835 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:29.231985 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.233421 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.236547 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.237990 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.238139 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.238398 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.241321 (+ 2923us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.242312 (+ 991us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":1233,"threads_started":1}
I20260929 15:48:29.243721 16347 tablet_service.cc:1467] Processing CreateTablet for tablet ddb2578bc5b1439093352339dc9f1fea (DEFAULT_TABLE table=test-table [id=a39b977fddd04b16b54dd144e3c851e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:29.244724 16213 tablet_service.cc:1467] Processing CreateTablet for tablet ddb2578bc5b1439093352339dc9f1fea (DEFAULT_TABLE table=test-table [id=a39b977fddd04b16b54dd144e3c851e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:29.245448 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb2578bc5b1439093352339dc9f1fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:29.246001 16213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb2578bc5b1439093352339dc9f1fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:29.246336 16079 tablet_service.cc:1467] Processing CreateTablet for tablet ddb2578bc5b1439093352339dc9f1fea (DEFAULT_TABLE table=test-table [id=a39b977fddd04b16b54dd144e3c851e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:29.248317 16079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb2578bc5b1439093352339dc9f1fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:29.268215 16432 tablet_bootstrap.cc:492] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Bootstrap starting.
I20260929 15:48:29.272784 16433 tablet_bootstrap.cc:492] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Bootstrap starting.
I20260929 15:48:29.273401 16432 tablet_bootstrap.cc:654] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:29.275056 16431 tablet_bootstrap.cc:492] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Bootstrap starting.
I20260929 15:48:29.275038 16432 log.cc:824] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:29.278394 16433 tablet_bootstrap.cc:654] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:29.279582 16433 log.cc:824] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:29.279938 16432 tablet_bootstrap.cc:492] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: No bootstrap required, opened a new log
I20260929 15:48:29.280256 16432 ts_tablet_manager.cc:1397] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260929 15:48:29.282823 16431 tablet_bootstrap.cc:654] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:29.284264 16431 log.cc:824] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:29.285503 16433 tablet_bootstrap.cc:492] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: No bootstrap required, opened a new log
I20260929 15:48:29.285984 16433 ts_tablet_manager.cc:1397] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260929 15:48:29.289328 16432 raft_consensus.cc:348] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.289788 16432 raft_consensus.cc:374] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:29.289892 16432 raft_consensus.cc:729] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7743f2466e4cb88c849b6d7af1230b, State: Initialized, Role: FOLLOWER
I20260929 15:48:29.290503 16432 consensus_queue.cc:260] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.291399 16431 tablet_bootstrap.cc:492] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: No bootstrap required, opened a new log
I20260929 15:48:29.291791 16412 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
I20260929 15:48:29.292393 16432 ts_tablet_manager.cc:1428] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260929 15:48:29.292737 16431 ts_tablet_manager.cc:1397] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260929 15:48:29.297873 16433 raft_consensus.cc:348] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.298422 16433 raft_consensus.cc:374] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:29.298520 16433 raft_consensus.cc:729] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6bbc005d6a468d9aed346a14b292fa, State: Initialized, Role: FOLLOWER
I20260929 15:48:29.299782 16433 consensus_queue.cc:260] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.302028 16433 ts_tablet_manager.cc:1428] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260929 15:48:29.303857 16431 raft_consensus.cc:348] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.304356 16431 raft_consensus.cc:374] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:29.304476 16431 raft_consensus.cc:729] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5038ba53e7f04ca782d2d05afb291742, State: Initialized, Role: FOLLOWER
I20260929 15:48:29.305071 16431 consensus_queue.cc:260] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.306229 16278 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
I20260929 15:48:29.306825 16431 ts_tablet_manager.cc:1428] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 15:48:29.307710 16279 tablet.cc:2369] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:48:29.386468 16413 tablet.cc:2369] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:48:29.427498 16438 raft_consensus.cc:482] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:48:29.427815 16438 raft_consensus.cc:504] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.429715 16438 leader_election.cc:288] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
I20260929 15:48:29.436910 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.430323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52015 (local address 127.0.97.2:42089)
0929 15:48:29.430534 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.430540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.431199 (+ 659us) server_negotiation.cc:407] Connection header received
0929 15:48:29.431455 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.431460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.431528 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.431627 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:29.432430 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.433149 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.435085 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.435885 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.436064 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.436170 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.436504 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.436641 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:48:29.437053 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.430308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42089 (local address 127.0.97.1:52015)
0929 15:48:29.431076 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.431093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:29.431292 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:29.431673 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:29.431680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:29.431853 (+ 173us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:29.432270 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.432281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.433300 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.433307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.434899 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.434912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.436124 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.436131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.436328 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.436352 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:48:29.436468 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:48:29.436615 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"spinlock_wait_cycles":25984,"thread_start_us":453,"threads_started":1}
I20260929 15:48:29.437433 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.430368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33095 (local address 127.0.97.3:34941)
0929 15:48:29.430566 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.430572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.431671 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 15:48:29.431893 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.431904 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.431969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.432094 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:29.432748 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.433521 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.435454 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.436392 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.436565 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.436698 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.437011 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.437155 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:48:29.437510 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.430709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34941 (local address 127.0.97.1:33095)
0929 15:48:29.431535 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.431552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:29.431725 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:29.432125 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:29.432130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:29.432160 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:29.432549 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.432559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.433702 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.433710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.435252 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:29.435266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.436640 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.436646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:29.436830 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.436854 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:48:29.436982 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 15:48:29.437142 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":472,"threads_started":1}
I20260929 15:48:29.438215 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb2578bc5b1439093352339dc9f1fea" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5038ba53e7f04ca782d2d05afb291742" is_pre_election: true
I20260929 15:48:29.438664 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb2578bc5b1439093352339dc9f1fea" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff7743f2466e4cb88c849b6d7af1230b" is_pre_election: true
I20260929 15:48:29.438870 16233 raft_consensus.cc:2454] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 0.
I20260929 15:48:29.439142 16367 raft_consensus.cc:2454] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 0.
I20260929 15:48:29.439697 16035 leader_election.cc:302] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa, 5038ba53e7f04ca782d2d05afb291742; no voters:
I20260929 15:48:29.440142 16438 raft_consensus.cc:2781] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:48:29.440346 16438 raft_consensus.cc:482] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:48:29.440483 16438 raft_consensus.cc:3037] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:29.442764 16438 raft_consensus.cc:504] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.443432 16438 leader_election.cc:288] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 election: Requested vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
I20260929 15:48:29.444093 16367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb2578bc5b1439093352339dc9f1fea" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff7743f2466e4cb88c849b6d7af1230b"
I20260929 15:48:29.444098 16233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb2578bc5b1439093352339dc9f1fea" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5038ba53e7f04ca782d2d05afb291742"
I20260929 15:48:29.444365 16367 raft_consensus.cc:3037] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:29.444355 16233 raft_consensus.cc:3037] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:29.447713 16367 raft_consensus.cc:2454] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 1.
I20260929 15:48:29.447717 16233 raft_consensus.cc:2454] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 1.
I20260929 15:48:29.448364 16035 leader_election.cc:302] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa, 5038ba53e7f04ca782d2d05afb291742; no voters:
I20260929 15:48:29.448724 16438 raft_consensus.cc:2781] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:29.449183 16438 raft_consensus.cc:686] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 1 LEADER]: Becoming Leader. State: Replica: 0a6bbc005d6a468d9aed346a14b292fa, State: Running, Role: LEADER
I20260929 15:48:29.449689 16438 consensus_queue.cc:237] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:29.455179 15955 catalog_manager.cc:5495] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a6bbc005d6a468d9aed346a14b292fa (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0a6bbc005d6a468d9aed346a14b292fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } health_report { overall_health: UNKNOWN } } }
W20260929 15:48:29.487470 16145 tablet.cc:2369] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:48:29.487946 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.481481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.97.1:34787)
0929 15:48:29.481756 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.481763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.481785 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:48:29.482021 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.482026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.482136 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.482323 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:48:29.482752 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.483529 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.485379 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.485823 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.485993 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.486560 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:48:29.487103 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:48:29.487318 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.487478 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.487595 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 15:48:29.497394 16233 raft_consensus.cc:1261] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Refusing update from remote peer 0a6bbc005d6a468d9aed346a14b292fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:29.497395 16367 raft_consensus.cc:1261] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Refusing update from remote peer 0a6bbc005d6a468d9aed346a14b292fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:29.498605 16438 consensus_queue.cc:1035] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:29.499044 16442 consensus_queue.cc:1035] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:29.528391 15955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:29.528707 15955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:29.532428 15955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Sending DeleteTablet for 3 replicas of tablet ddb2578bc5b1439093352339dc9f1fea
I20260929 15:48:29.534184 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16016
I20260929 15:48:29.534181 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb2578bc5b1439093352339dc9f1fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48960
I20260929 15:48:29.534262 16079 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb2578bc5b1439093352339dc9f1fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41106
I20260929 15:48:29.535239 16213 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb2578bc5b1439093352339dc9f1fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38444
I20260929 15:48:29.549731 16457 tablet_replica.cc:330] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: stopping tablet replica
I20260929 15:48:29.550329 16457 raft_consensus.cc:2229] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:48:29.551090 16457 raft_consensus.cc:2258] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:29.553083 16459 tablet_replica.cc:330] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: stopping tablet replica
I20260929 15:48:29.553587 16459 raft_consensus.cc:2229] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:29.554167 16459 raft_consensus.cc:2258] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:29.554924 16456 tablet_replica.cc:330] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: stopping tablet replica
I20260929 15:48:29.555361 16456 raft_consensus.cc:2229] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:29.555847 16456 raft_consensus.cc:2258] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:29.556945 16457 ts_tablet_manager.cc:1905] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:29.558080 16456 ts_tablet_manager.cc:1905] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:29.562151 16459 ts_tablet_manager.cc:1905] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:48:29 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:48:29.565507 16456 ts_tablet_manager.cc:1918] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:29.565734 16456 log.cc:1196] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/ddb2578bc5b1439093352339dc9f1fea
I20260929 15:48:29.566326 16456 ts_tablet_manager.cc:1939] T ddb2578bc5b1439093352339dc9f1fea P ff7743f2466e4cb88c849b6d7af1230b: Deleting consensus metadata
I20260929 15:48:29.568579 15941 catalog_manager.cc:4841] TS ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): tablet ddb2578bc5b1439093352339dc9f1fea (table test-table [id=a39b977fddd04b16b54dd144e3c851e9]) successfully deleted
I20260929 15:48:29.574328 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.535753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.97.1:34787)
0929 15:48:29.536018 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.536028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.552368 (+ 16340us) server_negotiation.cc:407] Connection header received
0929 15:48:29.552623 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.552632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.552759 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.552866 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:29.555003 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.556125 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.558281 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.559070 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.559176 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.565531 (+ 6355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:29.565550 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:29.565568 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:29.565617 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:29.570002 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.571195 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.571206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.571209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.571421 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.571976 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.571981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.571984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.572536 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:29.572853 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.573525 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.573840 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:48:29.574470 16459 ts_tablet_manager.cc:1918] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:29.574815 16459 log.cc:1196] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/ddb2578bc5b1439093352339dc9f1fea
I20260929 15:48:29.575773 16459 ts_tablet_manager.cc:1939] T ddb2578bc5b1439093352339dc9f1fea P 5038ba53e7f04ca782d2d05afb291742: Deleting consensus metadata
I20260929 15:48:29.576408 16139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:29.579267 15942 catalog_manager.cc:4841] TS 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): tablet ddb2578bc5b1439093352339dc9f1fea (table test-table [id=a39b977fddd04b16b54dd144e3c851e9]) successfully deleted
I20260929 15:48:29.684989 16457 ts_tablet_manager.cc:1918] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:29.685313 16457 log.cc:1196] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/ddb2578bc5b1439093352339dc9f1fea
I20260929 15:48:29.685851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16016
W20260929 15:48:29.706885 15942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34787 (error 108)
I20260929 15:48:29.707127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16150
I20260929 15:48:29.709295 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.708675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41124)
0929 15:48:29.708886 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.709060 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 15:48:29.709615 15942 catalog_manager.cc:4584] TS 0a6bbc005d6a468d9aed346a14b292fa (127.0.97.1:34787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddb2578bc5b1439093352339dc9f1fea: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Sep 29 15:48:29 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:48:29.726904 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.708082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.97.2:42089)
0929 15:48:29.708326 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.708333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.708356 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:48:29.708437 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.708442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.708532 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.708652 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:29.710511 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.711591 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.713708 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.714291 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.714453 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.719435 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:29.719454 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:29.719471 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:29.719530 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:29.722930 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.723976 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.723985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.723988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.724136 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.724666 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.724672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.724675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.725198 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:29.725759 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.726287 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.726559 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:48:29.727870 16273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:29.763466 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.762962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41128)
0929 15:48:29.763210 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.763305 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 15:48:29.764858 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.764370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41140)
0929 15:48:29.764640 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.764733 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 15:48:29.826520 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.825848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41150)
0929 15:48:29.826126 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.826256 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 15:48:29.827255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16150
I20260929 15:48:29.828629 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.828012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41152)
0929 15:48:29.828320 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.828410 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 15:48:29.846791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16284
Sep 29 15:48:29 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:48:29.866147 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.847681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.0.97.3:34941)
0929 15:48:29.847889 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:29.847893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:29.847913 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:48:29.847992 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:29.847996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:29.848058 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:29.848142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:29.849773 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.850553 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.852490 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:29.852924 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:29.853091 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:29.858523 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:29.858547 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:29.858569 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:29.858634 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:29.862354 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.863343 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.863351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.863354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.863542 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:29.864328 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:29.864332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:29.864334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:29.864768 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:29.865116 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:29.865499 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 15:48:29.865755 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:48:29.866866 16407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:29.910135 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.909515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41164)
0929 15:48:29.909770 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.909902 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 15:48:29.911373 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:29.910926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41168)
0929 15:48:29.911148 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:29.911236 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 15:48:29.960865 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16284
I20260929 15:48:29.981245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 15923
I20260929 15:48:29.982812 15985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:30.097092 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 15923
I20260929 15:48:30.097148 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:30.096522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:41180)
0929 15:48:30.096799 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:30.096925 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 15:48:50.120395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--webserver_interface=127.0.97.62
--webserver_port=44363
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:48:50.476437 16465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:50.476752 16465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:50.476929 16465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:50.484512 16465 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 15:48:50.484623 16465 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 15:48:50.484663 16465 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 15:48:50.484751 16465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:48:50.484833 16465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:50.484948 16465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:50.485024 16465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:48:50.485107 16465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:48:50.495558 16465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:50.496927 16465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:50.498324 16465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:50.522074 16471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:50.522188 16472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:50.522161 16474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:50.522886 16465 server_base.cc:1034] running on GCE node
I20260929 15:48:50.523859 16465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:50.525661 16465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:50.526909 16465 hybrid_clock.cc:648] HybridClock initialized: now 1790696930526842 us; error 43 us; skew 500 ppm
I20260929 15:48:50.527287 16465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:50 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:50.530957 16465 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:48:50.532681 16465 webserver.cc:460] Webserver started at http://127.0.97.62:44363/ using document root <none> and password file <none>
I20260929 15:48:50.533447 16465 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:50.533552 16465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:50.539827 16465 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:48:50.542871 16481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:50.544445 16465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:48:50.544658 16465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "4f05b0d205ac435d97f846775b5dd097"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "baf9c738deaf607d2e3c759692110d0b"
server_key_iv: "9c4db492dc4a1ca84d949edd910d1a18"
server_key_version: "encryptionkey@0"
I20260929 15:48:50.545819 16465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:50.581552 16465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:50.586107 16465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:50.586439 16465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:50.606355 16465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45651
I20260929 15:48:50.606379 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45651 every 8 connection(s)
I20260929 15:48:50.608004 16465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:48:50.615214 16533 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:48:50.617789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16465
I20260929 15:48:50.618790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34787
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45651
--webserver_port=35217
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:48:50.619238 16533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap starting.
I20260929 15:48:50.641686 16533 log.cc:824] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:50.654726 16533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:48:50.655347 16533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap complete.
I20260929 15:48:50.664825 16533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:50.665714 16533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f05b0d205ac435d97f846775b5dd097, State: Initialized, Role: FOLLOWER
I20260929 15:48:50.666404 16533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:50.666663 16533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:48:50.666800 16533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:48:50.666990 16533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:48:50.671118 16533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:50.671700 16533 leader_election.cc:302] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f05b0d205ac435d97f846775b5dd097; no voters:
I20260929 15:48:50.672411 16533 leader_election.cc:288] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:48:50.672554 16537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:48:50.673954 16537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 LEADER]: Becoming Leader. State: Replica: 4f05b0d205ac435d97f846775b5dd097, State: Running, Role: LEADER
I20260929 15:48:50.674592 16537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:48:50.674826 16533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:48:50.677585 16539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f05b0d205ac435d97f846775b5dd097. Latest consensus state: current_term: 2 leader_uuid: "4f05b0d205ac435d97f846775b5dd097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260929 15:48:50.677523 16538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f05b0d205ac435d97f846775b5dd097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260929 15:48:50.677997 16539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:50.678023 16538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: This master's current role is: LEADER
I20260929 15:48:50.688628 16544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:48:50.696981 16544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:48:50.699216 16544 catalog_manager.cc:1259] Loaded cluster ID: 62cf2d7d77b94e7b9825e65bee0dece4
I20260929 15:48:50.699430 16544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:48:50.703892 16544 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:48:50.705968 16544 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Loaded TSK: 0
I20260929 15:48:50.745988 16544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Generated new TSK 1
I20260929 15:48:50.746469 16544 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:48:50.997993 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:50.998390 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:50.998565 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:51.006220 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:51.006403 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:51.006584 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:48:51.017138 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35217
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:51.018555 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:51.019927 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:51.035050 16561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:51.035326 16560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:51.035902 16563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:51.036229 16535 server_base.cc:1034] running on GCE node
I20260929 15:48:51.036913 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:51.037955 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:51.039178 16535 hybrid_clock.cc:648] HybridClock initialized: now 1790696931039129 us; error 41 us; skew 500 ppm
I20260929 15:48:51.039506 16535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:51 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:51.042994 16535 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:48:51.044759 16535 webserver.cc:460] Webserver started at http://127.0.97.1:35217/ using document root <none> and password file <none>
I20260929 15:48:51.045686 16535 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:51.045845 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:51.053138 16535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:48:51.058014 16570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:51.059645 16535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 15:48:51.059823 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "0a6bbc005d6a468d9aed346a14b292fa"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "69d31b414367b4a066c95b6b8476dac6"
server_key_iv: "21027ef3f12d58d33cfd7c53249467f0"
server_key_version: "encryptionkey@0"
I20260929 15:48:51.061218 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:51.076743 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:51.082144 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:51.082437 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:51.083616 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:51.085852 16577 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 15:48:51.091609 16577 ts_tablet_manager.cc:1852] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 15:48:51.092708 16577 ts_tablet_manager.cc:1905] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:51.100036 16577 ts_tablet_manager.cc:1918] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:51.100368 16577 ts_tablet_manager.cc:1939] T ddb2578bc5b1439093352339dc9f1fea P 0a6bbc005d6a468d9aed346a14b292fa: Deleting consensus metadata
I20260929 15:48:51.100788 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 15:48:51.100940 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260929 15:48:51.101128 16535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:51.101229 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:51.139273 16535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34787
I20260929 15:48:51.139334 16683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34787 every 8 connection(s)
I20260929 15:48:51.140831 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:48:51.147612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16535
I20260929 15:48:51.148445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42089
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45651
--webserver_port=46473
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:48:51 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:51.167615 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:51.144141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:42083)
0929 15:48:51.145107 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:51.145162 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:51.147094 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:51.148956 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:51.148964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:51.149603 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:51.150143 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:51.150166 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.152628 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.152632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:51.153903 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:51.153915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.155332 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.155337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:51.155520 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:51.156265 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:51.156287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:51.158519 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:51.163248 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:51.163255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:51.163277 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:51.163727 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:51.164269 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:51.164273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:51.164275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:51.164433 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:51.165421 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:51.165427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:51.165756 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 15:48:51.166033 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 15:48:51.166425 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260929 15:48:51.167955 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:51.143918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42083 (local address 127.0.97.62:45651)
0929 15:48:51.144541 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:51.144551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:51.145308 (+ 757us) server_negotiation.cc:407] Connection header received
0929 15:48:51.147742 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:51.147763 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:51.148251 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:51.148700 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:51.150404 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.152460 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.154052 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.155123 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.155332 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:51.158809 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:51.158822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:51.158842 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:51.158897 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:51.163026 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:51.163873 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:51.163884 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:51.163893 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:51.164097 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:51.164548 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:51.164556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:51.164562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:51.165222 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:51.165419 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:51.166278 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 15:48:51.166713 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":289,"threads_started":1}
I20260929 15:48:51.170707 16684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260929 15:48:51.171028 16684 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:51.171731 16684 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260929 15:48:51.173992 16498 ts_manager.cc:194] Registered new tserver with Master: 0a6bbc005d6a468d9aed346a14b292fa (127.0.97.1:34787)
I20260929 15:48:51.175678 16498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42083
W20260929 15:48:51.489274 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:51.489635 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:51.489796 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:51.497303 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:51.497488 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:51.497671 16688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:48:51.507776 16688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46473
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:51.509286 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:51.510701 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:51.525144 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:51.525132 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:51.525221 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:51.525895 16688 server_base.cc:1034] running on GCE node
I20260929 15:48:51.526651 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:51.527637 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:51.528880 16688 hybrid_clock.cc:648] HybridClock initialized: now 1790696931528829 us; error 37 us; skew 500 ppm
I20260929 15:48:51.529295 16688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:51 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:51.532641 16688 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:48:51.534353 16688 webserver.cc:460] Webserver started at http://127.0.97.2:46473/ using document root <none> and password file <none>
I20260929 15:48:51.535075 16688 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:51.535250 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:51.540906 16688 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:48:51.544001 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:51.545480 16688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:48:51.545692 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "5038ba53e7f04ca782d2d05afb291742"
format_stamp: "Formatted at 2026-09-29 15:48:28 on dist-test-slave-7491"
server_key: "2313bcb4c461655ac7cc2929dce83741"
server_key_iv: "4c8559342322a7680480943c51baf32e"
server_key_version: "encryptionkey@0"
I20260929 15:48:51.546744 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:51.565498 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:51.570688 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:51.571030 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:51.572249 16688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:51.573951 16688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:51.574122 16688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:51.574268 16688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:51.574363 16688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:51.610647 16688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42089
I20260929 15:48:51.610649 16815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42089 every 8 connection(s)
I20260929 15:48:51.612637 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:48:51.617676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16688
I20260929 15:48:51.618458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34941
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45651
--webserver_port=36299
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:48:51 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:51.639371 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:51.615446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34433 (local address 127.0.97.62:45651)
0929 15:48:51.615632 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:51.615637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:51.616918 (+ 1281us) server_negotiation.cc:407] Connection header received
0929 15:48:51.619703 (+ 2785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:51.619709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:51.619824 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:51.619944 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:51.622128 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.623269 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.625944 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.626748 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.626892 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:51.631397 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:51.631407 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:51.631415 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:51.631469 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:51.635414 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:51.636460 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:51.636469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:51.636476 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:51.636654 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:51.637184 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:51.637193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:51.637199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:51.637625 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:51.637799 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:51.638603 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 15:48:51.638895 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:48:51.640249 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:51.615961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:34433)
0929 15:48:51.616742 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:51.616798 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:51.619366 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:51.620240 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:51.620253 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:51.621066 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:51.621895 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:51.621924 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.623508 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.623516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:51.625742 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:51.625773 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:51.627275 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:51.627284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:51.627670 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:51.628713 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:51.628753 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:51.631055 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:51.635646 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:51.635658 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:51.635675 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:51.636304 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:51.636811 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:51.636815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:51.636817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:51.637039 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:51.637791 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:51.637798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:51.638202 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 15:48:51.638447 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 15:48:51.638871 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":348,"threads_started":1}
I20260929 15:48:51.641993 16816 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260929 15:48:51.642340 16816 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:51.643184 16816 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260929 15:48:51.644868 16498 ts_manager.cc:194] Registered new tserver with Master: 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
I20260929 15:48:51.645794 16498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34433
W20260929 15:48:51.964735 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:48:51.965029 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:48:51.965214 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:48:51.972816 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:48:51.972936 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:48:51.973147 16819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:48:51.983515 16819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36299
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:48:51.984933 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:48:51.986243 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:48:51.999853 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:51.999949 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:48:51.999915 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:48:52.001214 16819 server_base.cc:1034] running on GCE node
I20260929 15:48:52.001931 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:48:52.002887 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:48:52.004082 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790696932004035 us; error 50 us; skew 500 ppm
I20260929 15:48:52.004433 16819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:48:52 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:48:52.007539 16819 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:48:52.009196 16819 webserver.cc:460] Webserver started at http://127.0.97.3:36299/ using document root <none> and password file <none>
I20260929 15:48:52.009928 16819 fs_manager.cc:359] Metadata directory not provided
I20260929 15:48:52.010093 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:48:52.015576 16819 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:48:52.018587 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:52.020081 16819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:48:52.020339 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "ff7743f2466e4cb88c849b6d7af1230b"
format_stamp: "Formatted at 2026-09-29 15:48:29 on dist-test-slave-7491"
server_key: "8a527628e3eaf7734994a583125de03a"
server_key_iv: "a59ceb614e0741fb63fa3b559d116894"
server_key_version: "encryptionkey@0"
I20260929 15:48:52.021378 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:48:52.046432 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:48:52.050874 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:48:52.051174 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:48:52.052425 16819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:48:52.054113 16819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:48:52.054263 16819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:52.054435 16819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:48:52.054554 16819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:48:52.092149 16819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34941
I20260929 15:48:52.092270 16946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34941 every 8 connection(s)
I20260929 15:48:52.093868 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:48:52.098927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16819
Sep 29 15:48:52 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:48:52.115743 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.099948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.0.97.62:45651)
0929 15:48:52.100454 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.100460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.101000 (+ 540us) server_negotiation.cc:407] Connection header received
0929 15:48:52.101189 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.101193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.101244 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.101321 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:52.103310 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.104097 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.105563 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.106033 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.106187 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.109230 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:52.109236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:52.109240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:52.109275 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:52.112699 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.113462 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.113469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.113472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.113611 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.114041 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.114045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.114048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.114435 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:52.114691 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.115087 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.115345 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":295,"threads_started":1}
I20260929 15:48:52.117388 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.096659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45995 (local address 127.0.97.62:45651)
0929 15:48:52.096838 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.096843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.097972 (+ 1129us) server_negotiation.cc:407] Connection header received
0929 15:48:52.100462 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.100471 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.100586 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.100697 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:52.102973 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.103833 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.105766 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.106302 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.106458 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.110687 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:52.110692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:52.110695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:52.110741 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:52.113585 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.114574 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.114580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.114582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.114743 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.115301 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.115308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.115312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.115697 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:52.115812 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.116820 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.117038 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:48:52.118605 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.097083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:45995)
0929 15:48:52.097802 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.097840 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:52.100046 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:52.101002 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:52.101017 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:52.101841 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:48:52.102743 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.102773 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.104055 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.104063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.105571 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.105585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.106584 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.106593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.106869 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.107527 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:48:52.107558 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:48:52.110436 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:48:52.113741 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:52.113752 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:52.113770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:52.114435 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:52.114868 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:48:52.114871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:48:52.114874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:48:52.115152 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:48:52.115845 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:48:52.115853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:48:52.116376 (+ 523us) client_negotiation.cc:769] Sending connection context
0929 15:48:52.116714 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 15:48:52.117288 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260929 15:48:52.120070 16947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45651
I20260929 15:48:52.120445 16947 heartbeater.cc:464] Registering TS with master...
I20260929 15:48:52.121170 16947 heartbeater.cc:511] Master 127.0.97.62:45651 requested a full tablet report, sending...
I20260929 15:48:52.122553 16498 ts_manager.cc:194] Registered new tserver with Master: ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941)
I20260929 15:48:52.123358 16498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45995
I20260929 15:48:52.131206 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:48:52.137969 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.132263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.0.97.62:45651)
0929 15:48:52.132436 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.132440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.132866 (+ 426us) server_negotiation.cc:407] Connection header received
0929 15:48:52.133060 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.133064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.133217 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.133354 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:48:52.133733 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.134456 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.136098 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.136571 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.136665 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.137218 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:48:52.137452 (+ 234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 15:48:52.137764 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55014: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 15:48:52.138140 16686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55014: token expired
W20260929 15:48:52.138720 16417 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:45651)
I20260929 15:48:52.153582 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.139423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.0.97.62:45651)
0929 15:48:52.139581 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.139585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.139599 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:48:52.139646 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.139649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.139708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.139783 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:52.141553 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.142297 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.144389 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.144789 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.144883 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.147551 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:52.147557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:52.147560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:52.147597 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:52.150351 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.151364 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.151370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.151374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.151510 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.152010 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.152022 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.152024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.152439 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:52.152697 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.153008 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.153248 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:48:52.157193 16498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:48:52.164844 16498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:48:52.179423 16684 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
I20260929 15:48:52.193235 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.177970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39150)
0929 15:48:52.184673 (+ 6703us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.184688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:52.184837 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:52.185577 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:52.185584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:52.185627 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:52.185958 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.185966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.188384 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.188391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.190694 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.190706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.192284 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.192291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.192541 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.192607 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 15:48:52.192744 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 15:48:52.192930 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6600,"mutex_wait_us":25,"thread_start_us":5708,"threads_started":1}
I20260929 15:48:52.194275 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.177625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34941 (local address 127.0.0.1:36108)
0929 15:48:52.179325 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.179364 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:52.179545 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:52.187116 (+ 7571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:52.187120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:52.187143 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:52.187529 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.187541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.189424 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.189432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.191499 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.191513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.193440 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.193445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.193613 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.193649 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:48:52.193747 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:48:52.193897 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"mutex_wait_us":51,"thread_start_us":1282,"threads_started":1}
I20260929 15:48:52.194325 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.178390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.97.3:34941)
0929 15:48:52.184128 (+ 5738us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.184138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.184199 (+ 61us) server_negotiation.cc:407] Connection header received
0929 15:48:52.186544 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.186557 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.186884 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.187727 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:52.188052 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.189277 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.191701 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.193207 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.193362 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.193657 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.193824 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.193988 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5530,"thread_start_us":5357,"threads_started":1}
I20260929 15:48:52.194772 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.178434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.0.97.1:34787)
0929 15:48:52.179150 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.179159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.184808 (+ 5649us) server_negotiation.cc:407] Connection header received
0929 15:48:52.184996 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.185008 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.185333 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.186148 (+ 815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:52.186314 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.188233 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.190851 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.192001 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.193967 (+ 1966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.194197 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.194276 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.194466 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":371,"threads_started":1}
I20260929 15:48:52.196267 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.184213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42089 (local address 127.0.0.1:57204)
0929 15:48:52.185326 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.185346 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:52.185457 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:52.187827 (+ 2370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:52.187832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:52.187877 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:52.188243 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.188254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.190324 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.190329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.192583 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.192598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.195414 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.195424 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.195600 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.195638 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:48:52.195735 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 15:48:52.195885 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":5,"thread_start_us":853,"threads_started":1}
I20260929 15:48:52.196403 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.184522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.97.2:42089)
0929 15:48:52.187304 (+ 2782us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.187313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.187340 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:48:52.187428 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.187435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.187622 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.188425 (+ 803us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:52.188665 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.190168 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.192776 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.195197 (+ 2421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.195336 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.195515 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.195862 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.196029 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2445,"thread_start_us":2329,"threads_started":1}
I20260929 15:48:52.196790 16619 tablet_service.cc:1467] Processing CreateTablet for tablet 3e79492029e3483fb87c11bf5fcb2f9d (DEFAULT_TABLE table=test-table [id=51ca64c6e6814df695b9805a687946fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:52.198361 16619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e79492029e3483fb87c11bf5fcb2f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:52.198756 16882 tablet_service.cc:1467] Processing CreateTablet for tablet 3e79492029e3483fb87c11bf5fcb2f9d (DEFAULT_TABLE table=test-table [id=51ca64c6e6814df695b9805a687946fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:52.200229 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e79492029e3483fb87c11bf5fcb2f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:52.200347 16751 tablet_service.cc:1467] Processing CreateTablet for tablet 3e79492029e3483fb87c11bf5fcb2f9d (DEFAULT_TABLE table=test-table [id=51ca64c6e6814df695b9805a687946fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:48:52.201597 16751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e79492029e3483fb87c11bf5fcb2f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:48:52.206228 16963 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Bootstrap starting.
I20260929 15:48:52.209429 16963 tablet_bootstrap.cc:654] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:52.210669 16963 log.cc:824] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:52.214047 16963 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: No bootstrap required, opened a new log
I20260929 15:48:52.214354 16963 ts_tablet_manager.cc:1397] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 15:48:52.216315 16965 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Bootstrap starting.
I20260929 15:48:52.220202 16965 tablet_bootstrap.cc:654] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:52.221660 16965 log.cc:824] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:52.221867 16966 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Bootstrap starting.
I20260929 15:48:52.224862 16963 raft_consensus.cc:348] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.225378 16963 raft_consensus.cc:374] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:52.225523 16963 raft_consensus.cc:729] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6bbc005d6a468d9aed346a14b292fa, State: Initialized, Role: FOLLOWER
I20260929 15:48:52.225694 16966 tablet_bootstrap.cc:654] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Neither blocks nor log segments found. Creating new log.
I20260929 15:48:52.225917 16965 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: No bootstrap required, opened a new log
I20260929 15:48:52.226233 16965 ts_tablet_manager.cc:1397] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 15:48:52.226293 16963 consensus_queue.cc:260] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.227002 16966 log.cc:824] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Log is configured to *not* fsync() on all Append() calls
I20260929 15:48:52.228144 16963 ts_tablet_manager.cc:1428] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260929 15:48:52.230473 16966 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: No bootstrap required, opened a new log
I20260929 15:48:52.230827 16966 ts_tablet_manager.cc:1397] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 15:48:52.237991 16965 raft_consensus.cc:348] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.238512 16965 raft_consensus.cc:374] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:52.238619 16965 raft_consensus.cc:729] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7743f2466e4cb88c849b6d7af1230b, State: Initialized, Role: FOLLOWER
I20260929 15:48:52.239287 16965 consensus_queue.cc:260] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.240572 16947 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
I20260929 15:48:52.241271 16965 ts_tablet_manager.cc:1428] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 15:48:52.241677 16966 raft_consensus.cc:348] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.242138 16966 raft_consensus.cc:374] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:48:52.242249 16966 raft_consensus.cc:729] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5038ba53e7f04ca782d2d05afb291742, State: Initialized, Role: FOLLOWER
I20260929 15:48:52.242980 16966 consensus_queue.cc:260] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.244400 16816 heartbeater.cc:503] Master 127.0.97.62:45651 was elected leader, sending a full tablet report...
I20260929 15:48:52.244921 16966 ts_tablet_manager.cc:1428] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260929 15:48:52.345754 16948 tablet.cc:2369] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:48:52.364849 16817 tablet.cc:2369] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:48:52.393551 16685 tablet.cc:2369] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:48:52.398113 16968 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:48:52.398424 16968 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.400200 16968 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
I20260929 15:48:52.407822 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.400844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37777 (local address 127.0.97.2:42089)
0929 15:48:52.401080 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.401088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.402151 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 15:48:52.402351 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.402355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.402414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.402528 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:52.403175 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.403876 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.405919 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.406719 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.406890 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.407001 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.407402 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.407537 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:48:52.407822 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.400683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34577 (local address 127.0.97.3:34941)
0929 15:48:52.400931 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.400940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.402077 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 15:48:52.402270 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.402284 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.402374 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.402445 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:48:52.403113 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.403876 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.405906 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.406720 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.406869 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.407005 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.407400 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.407541 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:48:52.408030 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.400638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34941 (local address 127.0.97.1:34577)
0929 15:48:52.401938 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.401954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:52.402090 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:52.402503 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:52.402511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:52.402536 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:52.402956 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.402968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.404074 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.404081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.405746 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.405758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.406983 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.407002 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.407235 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.407259 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:48:52.407371 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 15:48:52.407531 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":87,"thread_start_us":672,"threads_started":1}
I20260929 15:48:52.408419 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.400740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42089 (local address 127.0.97.1:37777)
0929 15:48:52.401994 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.402036 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:48:52.402205 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:48:52.402567 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:48:52.402572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:48:52.402603 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:48:52.402983 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.402991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.404013 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.404021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.405739 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:48:52.405751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.407024 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.407033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:48:52.407220 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.407255 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:48:52.407368 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 15:48:52.407540 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":94,"thread_start_us":802,"threads_started":1}
I20260929 15:48:52.409284 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e79492029e3483fb87c11bf5fcb2f9d" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff7743f2466e4cb88c849b6d7af1230b" is_pre_election: true
I20260929 15:48:52.409423 16771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e79492029e3483fb87c11bf5fcb2f9d" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5038ba53e7f04ca782d2d05afb291742" is_pre_election: true
I20260929 15:48:52.409919 16902 raft_consensus.cc:2454] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 0.
I20260929 15:48:52.409919 16771 raft_consensus.cc:2454] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 0.
I20260929 15:48:52.410705 16573 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa, 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b; no voters:
I20260929 15:48:52.411173 16968 raft_consensus.cc:2781] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:48:52.411334 16968 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:48:52.411466 16968 raft_consensus.cc:3037] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:52.414207 16968 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.414922 16968 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 election: Requested vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
I20260929 15:48:52.415606 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e79492029e3483fb87c11bf5fcb2f9d" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff7743f2466e4cb88c849b6d7af1230b"
I20260929 15:48:52.415606 16771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e79492029e3483fb87c11bf5fcb2f9d" candidate_uuid: "0a6bbc005d6a468d9aed346a14b292fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5038ba53e7f04ca782d2d05afb291742"
I20260929 15:48:52.415865 16902 raft_consensus.cc:3037] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:52.415854 16771 raft_consensus.cc:3037] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:48:52.419517 16771 raft_consensus.cc:2454] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 1.
I20260929 15:48:52.419517 16902 raft_consensus.cc:2454] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a6bbc005d6a468d9aed346a14b292fa in term 1.
I20260929 15:48:52.420151 16573 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa, 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b; no voters:
I20260929 15:48:52.420524 16968 raft_consensus.cc:2781] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:48:52.421037 16968 raft_consensus.cc:686] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 LEADER]: Becoming Leader. State: Replica: 0a6bbc005d6a468d9aed346a14b292fa, State: Running, Role: LEADER
I20260929 15:48:52.421541 16968 consensus_queue.cc:237] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:48:52.426591 16498 catalog_manager.cc:5495] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a6bbc005d6a468d9aed346a14b292fa (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0a6bbc005d6a468d9aed346a14b292fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } health_report { overall_health: UNKNOWN } } }
I20260929 15:48:52.491436 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.485275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.0.97.1:34787)
0929 15:48:52.485467 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.485471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.485490 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:48:52.485735 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.485740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.485833 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.485978 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:48:52.486396 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.487157 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.488966 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.489476 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.489619 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.490182 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:48:52.490730 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:48:52.490853 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.491011 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.491092 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 15:48:52.500741 16902 raft_consensus.cc:1261] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Refusing update from remote peer 0a6bbc005d6a468d9aed346a14b292fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:52.500739 16771 raft_consensus.cc:1261] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Refusing update from remote peer 0a6bbc005d6a468d9aed346a14b292fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:48:52.502198 16974 consensus_queue.cc:1035] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:52.502655 16968 consensus_queue.cc:1035] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:48:52.534979 16497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:52.535365 16497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:48:52.539454 16497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Sending DeleteTablet for 3 replicas of tablet 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:48:52.540931 16619 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e79492029e3483fb87c11bf5fcb2f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39150
I20260929 15:48:52.541070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16535
I20260929 15:48:52.541128 16751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e79492029e3483fb87c11bf5fcb2f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57204
I20260929 15:48:52.541146 16882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e79492029e3483fb87c11bf5fcb2f9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:48:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36108
I20260929 15:48:52.555649 16986 tablet_replica.cc:330] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: stopping tablet replica
I20260929 15:48:52.560532 16986 raft_consensus.cc:2229] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:48:52.561995 16986 raft_consensus.cc:2258] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:52.565944 16986 ts_tablet_manager.cc:1905] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:52.569249 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.542699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.0.97.1:34787)
0929 15:48:52.542942 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.542951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.542975 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:48:52.543049 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.543057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.543173 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.543293 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:52.545989 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.547605 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.550652 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.551571 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.551736 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.557004 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:52.557030 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:52.557057 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:52.557135 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:52.563058 (+ 5923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.564679 (+ 1621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.564694 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.564701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.565119 (+ 418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.565881 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.565891 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.565897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.566667 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:52.567958 (+ 1291us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.568245 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.568651 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:48:52.571385 16679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:52.571863 16988 tablet_replica.cc:330] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: stopping tablet replica
I20260929 15:48:52.572470 16988 raft_consensus.cc:2229] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:52.572743 16987 tablet_replica.cc:330] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: stopping tablet replica
I20260929 15:48:52.573156 16988 raft_consensus.cc:2258] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:52.573225 16987 raft_consensus.cc:2229] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:48:52.573807 16987 raft_consensus.cc:2258] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:48:52.576720 16988 ts_tablet_manager.cc:1905] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:52.576668 16987 ts_tablet_manager.cc:1905] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:48:52.585034 16988 ts_tablet_manager.cc:1918] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:52.585034 16987 ts_tablet_manager.cc:1918] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:48:52.585268 16988 log.cc:1196] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:48:52.585268 16987 log.cc:1196] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:48:52.585947 16988 ts_tablet_manager.cc:1939] T 3e79492029e3483fb87c11bf5fcb2f9d P ff7743f2466e4cb88c849b6d7af1230b: Deleting consensus metadata
I20260929 15:48:52.585950 16987 ts_tablet_manager.cc:1939] T 3e79492029e3483fb87c11bf5fcb2f9d P 5038ba53e7f04ca782d2d05afb291742: Deleting consensus metadata
I20260929 15:48:52.588369 16484 catalog_manager.cc:4841] TS ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): tablet 3e79492029e3483fb87c11bf5fcb2f9d (table test-table [id=51ca64c6e6814df695b9805a687946fb]) successfully deleted
I20260929 15:48:52.588414 16485 catalog_manager.cc:4841] TS 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): tablet 3e79492029e3483fb87c11bf5fcb2f9d (table test-table [id=51ca64c6e6814df695b9805a687946fb]) successfully deleted
I20260929 15:48:52.677650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16535
I20260929 15:48:52.696260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16688
W20260929 15:48:52.696339 16485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34787 (error 108)
I20260929 15:48:52.698616 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.697703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39180)
0929 15:48:52.698161 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.698300 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":218,"threads_started":1}
W20260929 15:48:52.699060 16485 catalog_manager.cc:4584] TS 0a6bbc005d6a468d9aed346a14b292fa (127.0.97.1:34787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e79492029e3483fb87c11bf5fcb2f9d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
I20260929 15:48:52.712843 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.697669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.0.97.2:42089)
0929 15:48:52.697904 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.697914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.697938 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:48:52.698016 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.698022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.698120 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.698253 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:52.699965 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.700829 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.702438 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.702889 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.703000 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.706046 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:52.706068 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:52.706090 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:52.706161 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:52.709332 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.710329 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.710338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.710340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.710503 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.710979 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.710984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.710986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.711443 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:52.711679 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.712155 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.712444 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:48:52.713678 16811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:52.752955 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.752366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39192)
0929 15:48:52.752609 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.752748 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 15:48:52.754227 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.753763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39196)
0929 15:48:52.753953 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.754048 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 15:48:52.808017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16688
I20260929 15:48:52.814936 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.814321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39198)
0929 15:48:52.814586 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.814731 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 15:48:52.816231 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.815683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39212)
0929 15:48:52.815889 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.815991 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:48:52.826005 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16819
I20260929 15:48:52.841866 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.827045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.0.97.3:34941)
0929 15:48:52.827223 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 15:48:52.827228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:48:52.827248 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:48:52.827385 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:48:52.827389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:48:52.827464 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:48:52.827553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:48:52.829176 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.829939 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.831430 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:48:52.832013 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:48:52.832145 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:48:52.835269 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:48:52.835286 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:48:52.835304 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:48:52.835347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:48:52.838617 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.839545 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.839553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.839555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.839721 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:48:52.840249 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:48:52.840253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:48:52.840256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:48:52.840639 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:48:52.840898 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:48:52.841276 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 15:48:52.841466 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:48:52.842620 16942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:52.897435 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.896826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39228)
0929 15:48:52.897110 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.897241 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 15:48:52.898768 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:52.898234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39238)
0929 15:48:52.898477 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:52.898571 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 15:48:52.937569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16819
I20260929 15:48:52.955180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 16465
I20260929 15:48:52.956416 16528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:48:53.073417 16991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:48:53.072648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34787 (local address 127.0.0.1:39244)
0929 15:48:53.072860 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 15:48:53.073141 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 15:48:53.073606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 16465
I20260929 15:49:13.096802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--webserver_interface=127.0.97.62
--webserver_port=44363
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:13.450088 16995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:13.450429 16995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:13.450598 16995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:13.458253 16995 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 15:49:13.458379 16995 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 15:49:13.458457 16995 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 15:49:13.458532 16995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:13.458635 16995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:13.458729 16995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:13.458820 16995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:13.458915 16995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:13.469439 16995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:13.470927 16995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:13.472386 16995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:13.485906 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:13.485848 17001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:13.486263 16995 server_base.cc:1034] running on GCE node
W20260929 15:49:13.485879 17003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:13.487530 16995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:13.489179 16995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:13.490466 16995 hybrid_clock.cc:648] HybridClock initialized: now 1790696953490379 us; error 64 us; skew 500 ppm
I20260929 15:49:13.490844 16995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:13 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:13.494143 16995 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:49:13.495779 16995 webserver.cc:460] Webserver started at http://127.0.97.62:44363/ using document root <none> and password file <none>
I20260929 15:49:13.496614 16995 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:13.496759 16995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:13.502588 16995 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 15:49:13.506623 17010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:13.508513 16995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:49:13.508754 16995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "4f05b0d205ac435d97f846775b5dd097"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "baf9c738deaf607d2e3c759692110d0b"
server_key_iv: "9c4db492dc4a1ca84d949edd910d1a18"
server_key_version: "encryptionkey@0"
I20260929 15:49:13.509830 16995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:13.528038 16995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:13.532639 16995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:13.533020 16995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:13.552891 16995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45651
I20260929 15:49:13.552990 17061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45651 every 8 connection(s)
I20260929 15:49:13.554551 16995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:13.562489 17062 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:49:13.564867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 16995
I20260929 15:49:13.566033 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34787
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45651
--webserver_port=35217
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:49:13.567286 17062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap starting.
I20260929 15:49:13.593199 17062 log.cc:824] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:13.611726 17062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:49:13.612501 17062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Bootstrap complete.
I20260929 15:49:13.622581 17062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:49:13.623541 17062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f05b0d205ac435d97f846775b5dd097, State: Initialized, Role: FOLLOWER
I20260929 15:49:13.624477 17062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:49:13.624749 17062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:13.624877 17062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:13.625078 17062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 2 FOLLOWER]: Advancing to term 3
I20260929 15:49:13.629356 17062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:49:13.629910 17062 leader_election.cc:302] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f05b0d205ac435d97f846775b5dd097; no voters:
I20260929 15:49:13.630631 17062 leader_election.cc:288] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 15:49:13.630782 17066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 15:49:13.632054 17066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [term 3 LEADER]: Becoming Leader. State: Replica: 4f05b0d205ac435d97f846775b5dd097, State: Running, Role: LEADER
I20260929 15:49:13.632735 17066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } }
I20260929 15:49:13.633044 17062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:13.635720 17067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4f05b0d205ac435d97f846775b5dd097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260929 15:49:13.636268 17067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:13.637014 17068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f05b0d205ac435d97f846775b5dd097. Latest consensus state: current_term: 3 leader_uuid: "4f05b0d205ac435d97f846775b5dd097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f05b0d205ac435d97f846775b5dd097" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45651 } } }
I20260929 15:49:13.637382 17068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:13.639375 17072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:13.650791 17072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:13.652909 17072 catalog_manager.cc:1259] Loaded cluster ID: 62cf2d7d77b94e7b9825e65bee0dece4
I20260929 15:49:13.653090 17072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:13.658878 17072 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:13.661600 17072 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Loaded TSK: 0
I20260929 15:49:13.661912 17072 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f05b0d205ac435d97f846775b5dd097: Loaded TSK: 1
F20260929 15:49:13.663563 17072 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb16988a9ad google::LogMessage::Fail() at ??:0
@ 0x7fb16988d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb16988a4de google::LogMessage::Flush() at ??:0
@ 0x7fb16988ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb16d5ee7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb16d5dd73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb17f2b91c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb17f2b8783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb17f3442a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb17f3440c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb17f38e5e9 std::function<>::operator()() at ??:0
@ 0x7fb17f2bccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb17f2bac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb17f342610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb17f342439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb16c4d6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb16c4d991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb16c4d9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb16c4b4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb16f78e6db start_thread at ??:0
@ 0x7fb16872461f clone at ??:0
W20260929 15:49:14.007562 17064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:14.007889 17064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:14.007988 17064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:14.015588 17064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:14.015720 17064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:14.015841 17064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:14.026412 17064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35217
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:14.027746 17064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:14.029170 17064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:14.042687 17109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:14.042694 17112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:14.042729 17110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:14.043618 17064 server_base.cc:1034] running on GCE node
I20260929 15:49:14.044364 17064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:14.045426 17064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:14.046691 17064 hybrid_clock.cc:648] HybridClock initialized: now 1790696954046621 us; error 67 us; skew 500 ppm
I20260929 15:49:14.047043 17064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:14 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:14.050261 17064 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:49:14.051885 17064 webserver.cc:460] Webserver started at http://127.0.97.1:35217/ using document root <none> and password file <none>
I20260929 15:49:14.052683 17064 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:14.052837 17064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:14.058652 17064 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 15:49:14.061990 17119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:14.063467 17064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:49:14.063664 17064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "0a6bbc005d6a468d9aed346a14b292fa"
format_stamp: "Formatted at 2026-09-29 15:48:27 on dist-test-slave-7491"
server_key: "69d31b414367b4a066c95b6b8476dac6"
server_key_iv: "21027ef3f12d58d33cfd7c53249467f0"
server_key_version: "encryptionkey@0"
I20260929 15:49:14.064824 17064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:14.104265 17064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:14.109011 17064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:14.109334 17064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:14.110417 17064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:14.112754 17126 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 15:49:14.119037 17064 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 15:49:14.119266 17064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260929 15:49:14.119481 17064 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 15:49:14.123230 17064 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 15:49:14.123386 17064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20260929 15:49:14.123855 17126 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Bootstrap starting.
I20260929 15:49:14.133848 17126 log.cc:824] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:14.149049 17126 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:49:14.150287 17126 tablet_bootstrap.cc:492] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Bootstrap complete.
I20260929 15:49:14.150951 17126 ts_tablet_manager.cc:1397] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260929 15:49:14.163058 17126 raft_consensus.cc:348] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:14.165408 17126 raft_consensus.cc:729] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6bbc005d6a468d9aed346a14b292fa, State: Initialized, Role: FOLLOWER
I20260929 15:49:14.166666 17126 consensus_queue.cc:260] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:14.168895 17126 ts_tablet_manager.cc:1428] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa: Time spent starting tablet: real 0.018s user 0.017s sys 0.002s
I20260929 15:49:14.172626 17064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34787
I20260929 15:49:14.172659 17234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34787 every 8 connection(s)
I20260929 15:49:14.173988 17064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:14.178252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17064
I20260929 15:49:14.178956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42089
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45651
--webserver_port=46473
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:49:14.179598 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:14.177184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:37651)
0929 15:49:14.178225 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:14.178521 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":374,"threads_started":1}
W20260929 15:49:14.180425 17235 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45651 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45651: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
I20260929 15:49:14.181448 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:14.181045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:59257)
0929 15:49:14.181227 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:14.181326 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:49:14.182626 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:14.182088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:45911)
0929 15:49:14.182313 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:14.182426 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 15:49:14.183071 17235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:49:14.532088 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:14.532511 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:14.532683 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:14.540376 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:14.540551 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:14.540735 17238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:14.550905 17238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46473
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:14.552362 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:14.553704 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:14.567718 17243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:14.567669 17244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:14.568509 17246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:14.569001 17238 server_base.cc:1034] running on GCE node
I20260929 15:49:14.569796 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:14.570847 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:14.572086 17238 hybrid_clock.cc:648] HybridClock initialized: now 1790696954572049 us; error 55 us; skew 500 ppm
I20260929 15:49:14.572461 17238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:14 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:14.582947 17238 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:49:14.584772 17238 webserver.cc:460] Webserver started at http://127.0.97.2:46473/ using document root <none> and password file <none>
I20260929 15:49:14.585557 17238 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:14.585737 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:14.591727 17238 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:49:14.594837 17253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:14.596396 17238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:49:14.596606 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "5038ba53e7f04ca782d2d05afb291742"
format_stamp: "Formatted at 2026-09-29 15:48:28 on dist-test-slave-7491"
server_key: "2313bcb4c461655ac7cc2929dce83741"
server_key_iv: "4c8559342322a7680480943c51baf32e"
server_key_version: "encryptionkey@0"
I20260929 15:49:14.597741 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:14.621560 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:14.625793 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:14.626096 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:14.627120 17238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:14.628583 17238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:14.628731 17238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:14.628863 17238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:14.628953 17238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:14.664132 17238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42089
I20260929 15:49:14.664296 17365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42089 every 8 connection(s)
I20260929 15:49:14.665735 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:14.666558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17238
I20260929 15:49:14.667400 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34941
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45651
--webserver_port=36299
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:49:14.671600 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:14.669317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:55327)
0929 15:49:14.670304 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:14.670599 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":461,"threads_started":1}
W20260929 15:49:14.672506 17366 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45651 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45651: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
I20260929 15:49:14.673496 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:14.673108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:47189)
0929 15:49:14.673287 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:14.673375 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:49:14.674549 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:14.674202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:50695)
0929 15:49:14.674359 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:14.674434 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 15:49:14.674923 17366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:49:15.011929 17369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:15.012279 17369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:15.012493 17369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:15.020121 17369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:15.020295 17369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:15.020505 17369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:15.031090 17369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36299
--tserver_master_addrs=127.0.97.62:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:15.032752 17369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:15.034183 17369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:15.048666 17375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:15.048784 17374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:15.048816 17377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:15.050048 17369 server_base.cc:1034] running on GCE node
I20260929 15:49:15.050741 17369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:15.051836 17369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:15.053076 17369 hybrid_clock.cc:648] HybridClock initialized: now 1790696955052999 us; error 82 us; skew 500 ppm
I20260929 15:49:15.053442 17369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:15 dist-test-slave-7491 krb5kdc[15910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:15.056914 17369 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:49:15.058691 17369 webserver.cc:460] Webserver started at http://127.0.97.3:36299/ using document root <none> and password file <none>
I20260929 15:49:15.059464 17369 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:15.059631 17369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:15.065773 17369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:49:15.069139 17384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:15.070609 17369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:49:15.070814 17369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "ff7743f2466e4cb88c849b6d7af1230b"
format_stamp: "Formatted at 2026-09-29 15:48:29 on dist-test-slave-7491"
server_key: "8a527628e3eaf7734994a583125de03a"
server_key_iv: "a59ceb614e0741fb63fa3b559d116894"
server_key_version: "encryptionkey@0"
I20260929 15:49:15.071873 17369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:15.102376 17369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:15.107527 17369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:15.107892 17369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:15.108997 17369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:15.110440 17369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:15.110570 17369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:15.110708 17369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:15.110796 17369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:15.146839 17369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34941
I20260929 15:49:15.146899 17496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34941 every 8 connection(s)
I20260929 15:49:15.148424 17369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:15.153400 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:15.151434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:58583)
0929 15:49:15.152270 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:15.152502 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":301,"threads_started":1}
W20260929 15:49:15.154043 17497 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45651 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45651: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
I20260929 15:49:15.155035 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:15.154614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:55751)
0929 15:49:15.154836 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:15.154913 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 15:49:15.155314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17369
I20260929 15:49:15.156049 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:15.155686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:49373)
0929 15:49:15.155842 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:15.155931 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 15:49:15.156374 17497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 15:49:15.184494 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:15.183656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:58355)
0929 15:49:15.184105 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:15.184269 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260929 15:49:15.676636 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:15.675607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:39109)
0929 15:49:15.676281 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:15.676384 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":345,"threads_started":1}
I20260929 15:49:16.016601 17503 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:16.016943 17503 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:16.018625 17503 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
Sep 29 15:49:16 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 29 15:49:16 dist-test-slave-7491 krb5kdc[15910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:49:16.053721 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.019493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44121 (local address 127.0.97.2:42089)
0929 15:49:16.032441 (+ 12948us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:16.032456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:16.032961 (+ 505us) server_negotiation.cc:407] Connection header received
0929 15:49:16.034640 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:16.034648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:16.034984 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:16.035369 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:16.037845 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.039031 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.040896 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.041735 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.041951 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:16.045883 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:16.045891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:16.045906 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:16.045949 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:16.049536 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:16.050539 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:16.050547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:16.050553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:16.050726 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:16.051208 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:16.051213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:16.051215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:16.051786 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:16.051993 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:16.052904 (+ 911us) server_negotiation.cc:299] Negotiation successful
0929 15:49:16.053293 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12775,"thread_start_us":12555,"threads_started":1}
I20260929 15:49:16.053737 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.018941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34941 (local address 127.0.97.1:53841)
0929 15:49:16.031943 (+ 13002us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:16.032020 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:16.032823 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:16.035522 (+ 2699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:16.035535 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:16.036675 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:16.037617 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:16.037643 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.039346 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.039351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:16.040732 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:16.040747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.041798 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.041804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:16.042092 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:16.042968 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:16.042986 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:16.045767 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:16.049711 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:16.049724 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:16.049736 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:16.050365 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:16.050853 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:16.050858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:16.050861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:16.051089 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:16.051930 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:16.051941 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:16.052374 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 15:49:16.052654 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 15:49:16.053254 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12805,"mutex_wait_us":175,"spinlock_wait_cycles":19712,"thread_start_us":12639,"threads_started":1}
I20260929 15:49:16.053728 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.019164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53841 (local address 127.0.97.3:34941)
0929 15:49:16.031502 (+ 12338us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:16.031529 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:16.032375 (+ 846us) server_negotiation.cc:407] Connection header received
0929 15:49:16.034543 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:16.034555 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:16.034981 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:16.035369 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:16.037844 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.039154 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.040895 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.041574 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.041822 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:16.046055 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:16.046063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:16.046076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:16.046117 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:16.049526 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:16.050539 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:16.050547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:16.050553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:16.050731 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:16.051211 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:16.051218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:16.051221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:16.051779 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:16.051992 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:16.052908 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 15:49:16.053280 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12049,"thread_start_us":11898,"threads_started":1}
I20260929 15:49:16.054227 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.019055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42089 (local address 127.0.97.1:44121)
0929 15:49:16.032810 (+ 13755us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:16.032825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:16.032984 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:16.035523 (+ 2539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:16.035535 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:16.036690 (+ 1155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:16.037618 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:16.037643 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.039240 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.039246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:16.040733 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:16.040748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:16.041952 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:16.041959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:16.042139 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:16.042920 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:16.042944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:16.045531 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:16.049709 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:16.049720 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:16.049736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:16.050367 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:16.050848 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:16.050853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:16.050857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:16.051074 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:16.051932 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:16.051939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:16.052373 (+ 434us) client_negotiation.cc:769] Sending connection context
0929 15:49:16.052654 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 15:49:16.053440 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13636,"mutex_wait_us":347,"spinlock_wait_cycles":48512,"thread_start_us":13508,"threads_started":1}
W20260929 15:49:16.057308 17122 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
W20260929 15:49:16.057612 17123 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:49:16.057760 17123 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa; no voters: 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b
I20260929 15:49:16.058104 17503 raft_consensus.cc:2726] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:49:16.158010 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.156998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:52725)
0929 15:49:16.157546 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:16.157713 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":261,"threads_started":1}
I20260929 15:49:16.185940 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.185486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:48469)
0929 15:49:16.185691 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:16.185791 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 15:49:16.678597 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:16.677781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:36353)
0929 15:49:16.678324 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:16.678418 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":315,"threads_started":1}
I20260929 15:49:17.160027 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:17.159212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:48297)
0929 15:49:17.159746 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:17.159850 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":314,"threads_started":1}
I20260929 15:49:17.187810 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:17.186936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:56183)
0929 15:49:17.187512 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:17.187605 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":341,"threads_started":1}
I20260929 15:49:17.611476 17512 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:17.611694 17512 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:17.612668 17512 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
W20260929 15:49:17.613742 17122 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
W20260929 15:49:17.614007 17123 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:49:17.614271 17123 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa; no voters: 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b
I20260929 15:49:17.614621 17512 raft_consensus.cc:2726] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:49:17.680591 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:17.679708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:44747)
0929 15:49:17.680303 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:17.680398 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":249,"threads_started":1}
I20260929 15:49:18.161775 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:18.161027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:35865)
0929 15:49:18.161509 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:18.161597 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260929 15:49:18.189797 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:18.189026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:36461)
0929 15:49:18.189530 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:18.189621 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260929 15:49:18.682457 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:18.681599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:35089)
0929 15:49:18.682150 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:18.682247 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260929 15:49:19.163702 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:19.162886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:42109)
0929 15:49:19.163430 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:19.163517 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":296,"threads_started":1}
I20260929 15:49:19.192193 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:19.191031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:59707)
0929 15:49:19.191803 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:19.191897 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":290,"threads_started":1}
I20260929 15:49:19.684623 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:19.683729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:47657)
0929 15:49:19.684353 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:19.684448 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":305,"threads_started":1}
I20260929 15:49:20.165549 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:20.164792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:57161)
0929 15:49:20.165302 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:20.165390 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":280,"threads_started":1}
I20260929 15:49:20.194737 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:20.193710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:56913)
0929 15:49:20.194365 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:20.194501 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":330,"threads_started":1}
I20260929 15:49:20.686633 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:20.685822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:40575)
0929 15:49:20.686352 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:20.686441 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":290,"threads_started":1}
I20260929 15:49:20.904048 17523 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:20.904366 17523 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:20.905182 17523 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
W20260929 15:49:20.906222 17122 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
W20260929 15:49:20.906424 17123 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:49:20.906517 17123 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa; no voters: 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b
I20260929 15:49:20.906854 17523 raft_consensus.cc:2726] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:49:21.167416 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:21.166711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:60929)
0929 15:49:21.167165 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:21.167242 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
I20260929 15:49:21.197160 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:21.196294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:52265)
0929 15:49:21.196840 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:21.196936 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":292,"threads_started":1}
I20260929 15:49:21.688823 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:21.687734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:36809)
0929 15:49:21.688521 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:21.688615 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":284,"threads_started":1}
I20260929 15:49:22.169538 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:22.168544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:34719)
0929 15:49:22.169245 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:22.169342 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":299,"threads_started":1}
I20260929 15:49:22.198868 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:22.198024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:51983)
0929 15:49:22.198508 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:22.198595 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
I20260929 15:49:22.690902 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:22.690106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:44861)
0929 15:49:22.690601 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:22.690716 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260929 15:49:23.171213 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:23.170464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:44995)
0929 15:49:23.170921 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:23.171005 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260929 15:49:23.200794 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:23.199922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:36419)
0929 15:49:23.200500 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:23.200591 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":349,"threads_started":1}
I20260929 15:49:23.692788 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:23.691936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:47735)
0929 15:49:23.692528 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:23.692621 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":348,"threads_started":1}
I20260929 15:49:24.161625 17533 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:24.161852 17533 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:24.162503 17533 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
W20260929 15:49:24.163498 17123 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
W20260929 15:49:24.163734 17122 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:49:24.163905 17122 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa; no voters: 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b
I20260929 15:49:24.164243 17533 raft_consensus.cc:2726] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:49:24.172987 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:24.172118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:50411)
0929 15:49:24.172754 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:24.172842 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":379,"threads_started":1}
I20260929 15:49:24.202836 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:24.201970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:54431)
0929 15:49:24.202476 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:24.202581 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":258,"threads_started":1}
I20260929 15:49:24.694810 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:24.693911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:60415)
0929 15:49:24.694514 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:24.694617 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":353,"threads_started":1}
I20260929 15:49:25.174906 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:25.174111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:35787)
0929 15:49:25.174619 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:25.174712 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":263,"threads_started":1}
I20260929 15:49:25.204773 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:25.203937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:52915)
0929 15:49:25.204476 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:25.204568 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":326,"threads_started":1}
I20260929 15:49:25.696718 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:25.695906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:44389)
0929 15:49:25.696444 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:25.696532 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
I20260929 15:49:26.176934 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:26.176007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:56427)
0929 15:49:26.176594 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:26.176720 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":327,"threads_started":1}
I20260929 15:49:26.206449 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:26.205706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:51925)
0929 15:49:26.206181 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:26.206272 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
I20260929 15:49:26.698483 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:26.697770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:57517)
0929 15:49:26.698219 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:26.698304 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260929 15:49:27.178843 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:27.177899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:45737)
0929 15:49:27.178488 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:27.178600 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":315,"threads_started":1}
I20260929 15:49:27.208437 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:27.207529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:40621)
0929 15:49:27.208060 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:27.208235 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260929 15:49:27.700572 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:27.699616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:58725)
0929 15:49:27.700234 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:27.700349 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260929 15:49:28.003592 17546 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:28.003851 17546 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:28.004642 17546 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
W20260929 15:49:28.005640 17122 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
W20260929 15:49:28.005815 17123 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:49:28.005934 17123 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa; no voters: 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b
I20260929 15:49:28.006213 17546 raft_consensus.cc:2726] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:49:28.180650 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:28.179794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:52837)
0929 15:49:28.180364 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:28.180461 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":262,"threads_started":1}
I20260929 15:49:28.210350 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:28.209470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:46231)
0929 15:49:28.210032 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:28.210118 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":286,"threads_started":1}
I20260929 15:49:28.702353 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:28.701551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:43293)
0929 15:49:28.702043 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:28.702135 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260929 15:49:29.182346 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:29.181599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:59329)
0929 15:49:29.182064 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:29.182155 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260929 15:49:29.212695 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:29.211644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.1:42103)
0929 15:49:29.212139 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:29.212295 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":249,"threads_started":1}
I20260929 15:49:29.704433 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:29.703582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.2:41091)
0929 15:49:29.704087 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:29.704269 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":251,"threads_started":1}
I20260929 15:49:30.086691 17553 raft_consensus.cc:482] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:30.086989 17553 raft_consensus.cc:504] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff7743f2466e4cb88c849b6d7af1230b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34941 } } peers { permanent_uuid: "0a6bbc005d6a468d9aed346a14b292fa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34787 } } peers { permanent_uuid: "5038ba53e7f04ca782d2d05afb291742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42089 } }
I20260929 15:49:30.087855 17553 leader_election.cc:288] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941), 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089)
W20260929 15:49:30.089066 17123 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5038ba53e7f04ca782d2d05afb291742 (127.0.97.2:42089): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
W20260929 15:49:30.089279 17122 leader_election.cc:341] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff7743f2466e4cb88c849b6d7af1230b (127.0.97.3:34941): Not found: Tablet not found: 3e79492029e3483fb87c11bf5fcb2f9d
I20260929 15:49:30.089407 17122 leader_election.cc:302] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a6bbc005d6a468d9aed346a14b292fa; no voters: 5038ba53e7f04ca782d2d05afb291742, ff7743f2466e4cb88c849b6d7af1230b
I20260929 15:49:30.089944 17553 raft_consensus.cc:2726] T 3e79492029e3483fb87c11bf5fcb2f9d P 0a6bbc005d6a468d9aed346a14b292fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:49:30.160038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17064
I20260929 15:49:30.180053 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:30.161548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.97.1:34787)
0929 15:49:30.162531 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:30.162541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:30.162580 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:49:30.162671 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:30.162680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:30.163007 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:30.163258 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:30.165114 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:30.166218 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:30.167807 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:30.168589 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:30.168748 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:30.171649 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:30.171663 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:30.171690 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:30.171744 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:30.175747 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:30.177179 (+ 1432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:30.177189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:30.177194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:30.177404 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:30.177992 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:30.177998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:30.178001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:30.178582 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:30.178912 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:30.179366 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 15:49:30.179639 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":436,"threads_started":1}
I20260929 15:49:30.181675 17230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:30.185732 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:30.184160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45651 (local address 127.0.97.3:38405)
0929 15:49:30.185228 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:30.185423 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":393,"threads_started":1}
I20260929 15:49:30.279572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17064
I20260929 15:49:30.298849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17238
I20260929 15:49:30.316955 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:30.300046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.0.97.2:42089)
0929 15:49:30.300770 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:30.300776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:30.300795 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:49:30.300876 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:30.300881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:30.300954 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:30.301111 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:30.302842 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:30.303812 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:30.305558 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:30.306136 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:30.306319 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:30.309994 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:30.310001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:30.310005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:30.310069 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:30.313718 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:30.314641 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:30.314646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:30.314651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:30.314814 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:30.315281 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:30.315286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:30.315288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:30.315609 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:30.315935 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:30.316269 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 15:49:30.316531 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":401,"threads_started":1}
I20260929 15:49:30.317847 17361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:30.407223 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17238
I20260929 15:49:30.429088 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17369
I20260929 15:49:30.446437 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:30.430581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.0.97.3:34941)
0929 15:49:30.431387 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:30.431392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:30.431412 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:49:30.431473 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:30.431477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:30.431555 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:30.431694 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:30.433282 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:30.434068 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:30.435816 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:30.436543 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:30.436747 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:30.439889 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:30.439896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:30.439899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:30.439943 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:30.443109 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:30.443949 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:30.443955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:30.443959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:30.444113 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:30.444836 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:30.444844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:30.444848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:30.445160 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:30.445459 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:30.445756 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 15:49:30.446019 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":440,"threads_started":1}
I20260929 15:49:30.447311 17492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:30.537396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17369
2026-09-29T15:49:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65159 ms)
[----------] 30 tests from SecurityITest (324720 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:49:30 dist-test-slave-7491 krb5kdc[17564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:49:30 dist-test-slave-7491 krb5kdc[17564](info): set up 2 sockets
Sep 29 15:49:30 dist-test-slave-7491 krb5kdc[17564](info): commencing operation
krb5kdc: starting...
W20260929 15:49:31.704517 388 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:49:31 dist-test-slave-7491 krb5kdc[17564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:49:31.995877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38995
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:32.379645 17577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:32.380070 17577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:32.380259 17577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:32.388904 17577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:32.389132 17577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:32.389312 17577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:32.389389 17577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:32.389427 17577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:32.401113 17577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:32.402591 17577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:32.403923 17577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:32.418313 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:32.418325 17583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:32.418174 17585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:32.419799 17577 server_base.cc:1034] running on GCE node
I20260929 15:49:32.420859 17577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:32.422465 17577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:32.423713 17577 hybrid_clock.cc:648] HybridClock initialized: now 1790696972423640 us; error 64 us; skew 500 ppm
I20260929 15:49:32.424207 17577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:32 dist-test-slave-7491 krb5kdc[17564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:32.433115 17577 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:49:32.434979 17577 webserver.cc:460] Webserver started at http://127.0.97.62:38709/ using document root <none> and password file <none>
I20260929 15:49:32.436498 17577 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:32.436683 17577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:32.437044 17577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:32.440531 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "2f56555fdbc44957802a225c6c15079e"
format_stamp: "Formatted at 2026-09-29 15:49:32 on dist-test-slave-7491"
server_key: "2c0bb0cd0439b88943355434d7ca15e7"
server_key_iv: "14d6397a2d020491aa8e9d7ff3aa85e7"
server_key_version: "encryptionkey@0"
I20260929 15:49:32.441412 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f56555fdbc44957802a225c6c15079e"
format_stamp: "Formatted at 2026-09-29 15:49:32 on dist-test-slave-7491"
server_key: "2c0bb0cd0439b88943355434d7ca15e7"
server_key_iv: "14d6397a2d020491aa8e9d7ff3aa85e7"
server_key_version: "encryptionkey@0"
I20260929 15:49:32.447965 17577 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:49:32.454303 17592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:32.455816 17577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:49:32.456037 17577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "2f56555fdbc44957802a225c6c15079e"
format_stamp: "Formatted at 2026-09-29 15:49:32 on dist-test-slave-7491"
server_key: "2c0bb0cd0439b88943355434d7ca15e7"
server_key_iv: "14d6397a2d020491aa8e9d7ff3aa85e7"
server_key_version: "encryptionkey@0"
I20260929 15:49:32.456310 17577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:32.484236 17577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:32.490653 17577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:32.491005 17577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:32.515321 17577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38995
I20260929 15:49:32.515362 17643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38995 every 8 connection(s)
I20260929 15:49:32.516929 17577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:32.522259 17644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:32.525182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17577
I20260929 15:49:32.525416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:32.526325 388 external_mini_cluster.cc:1428] Setting key 06219ae72e1392a3691f7e1efde03fcd
I20260929 15:49:32.537478 17644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e: Bootstrap starting.
I20260929 15:49:32.541865 17644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:32.543401 17644 log.cc:824] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:49:32 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:32.547238 17644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e: No bootstrap required, opened a new log
I20260929 15:49:32.557873 17644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f56555fdbc44957802a225c6c15079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260929 15:49:32.558271 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:32.528830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.0.97.62:38995)
0929 15:49:32.529706 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:32.529719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:32.529762 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:49:32.532734 (+ 2972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:32.532763 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:32.533258 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:32.533658 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:32.535795 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:32.537801 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:32.539905 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:32.540801 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:32.540971 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:32.546305 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:32.546315 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:32.546332 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:32.546384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:32.552134 (+ 5750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:32.553447 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:32.553457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:32.553467 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:32.553799 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:32.554413 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:32.554421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:32.554424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:32.555395 (+ 971us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:32.555724 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:32.556513 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 15:49:32.557129 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":407,"threads_started":1}
I20260929 15:49:32.558336 17644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:32.558455 17644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f56555fdbc44957802a225c6c15079e, State: Initialized, Role: FOLLOWER
I20260929 15:49:32.559255 17644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f56555fdbc44957802a225c6c15079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260929 15:49:32.559573 17644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:32.559722 17644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:32.559906 17644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:32.562853 17644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f56555fdbc44957802a225c6c15079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260929 15:49:32.563431 17644 leader_election.cc:302] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f56555fdbc44957802a225c6c15079e; no voters:
I20260929 15:49:32.564126 17644 leader_election.cc:288] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:32.564283 17649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:32.565025 17649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [term 1 LEADER]: Becoming Leader. State: Replica: 2f56555fdbc44957802a225c6c15079e, State: Running, Role: LEADER
I20260929 15:49:32.565649 17649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f56555fdbc44957802a225c6c15079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } }
I20260929 15:49:32.566447 17644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:32.568845 17650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f56555fdbc44957802a225c6c15079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f56555fdbc44957802a225c6c15079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } } }
I20260929 15:49:32.568984 17651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f56555fdbc44957802a225c6c15079e. Latest consensus state: current_term: 1 leader_uuid: "2f56555fdbc44957802a225c6c15079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f56555fdbc44957802a225c6c15079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38995 } } }
I20260929 15:49:32.569373 17650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:32.569370 17651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:32.570433 17656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:32.578132 17656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:32.592298 17656 catalog_manager.cc:1347] Generated new cluster ID: ece917eff62e42d0a34089ea458b3ad3
I20260929 15:49:32.592485 17656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:32.616976 17656 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:32.618439 17656 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:32.627449 17656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e: Generated new TSK 0
I20260929 15:49:32.628500 17656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:49:32.704675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38995
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:33.089815 17670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:33.090289 17670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:33.090415 17670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:33.099328 17670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:33.099473 17670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:33.099591 17670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:33.110158 17670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:33.111521 17670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:33.112931 17670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:33.127485 17676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:33.127478 17675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:33.128379 17678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:33.129289 17670 server_base.cc:1034] running on GCE node
I20260929 15:49:33.129976 17670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:33.131083 17670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:33.132339 17670 hybrid_clock.cc:648] HybridClock initialized: now 1790696973132278 us; error 43 us; skew 500 ppm
I20260929 15:49:33.132884 17670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:33 dist-test-slave-7491 krb5kdc[17564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:33.136771 17670 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:49:33.138756 17670 webserver.cc:460] Webserver started at http://127.0.97.1:42597/ using document root <none> and password file <none>
I20260929 15:49:33.139623 17670 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:33.139778 17670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:33.140146 17670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:33.143545 17670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "5e7c9f7fd65742448dcb279f61823456"
format_stamp: "Formatted at 2026-09-29 15:49:33 on dist-test-slave-7491"
server_key: "0e3c369249dcb782c52483308f675b9e"
server_key_iv: "2636ea00b91de0b344ce6bff460bc9f4"
server_key_version: "encryptionkey@0"
I20260929 15:49:33.144558 17670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e7c9f7fd65742448dcb279f61823456"
format_stamp: "Formatted at 2026-09-29 15:49:33 on dist-test-slave-7491"
server_key: "0e3c369249dcb782c52483308f675b9e"
server_key_iv: "2636ea00b91de0b344ce6bff460bc9f4"
server_key_version: "encryptionkey@0"
I20260929 15:49:33.151350 17670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:49:33.155794 17685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:33.157584 17670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:49:33.157799 17670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "5e7c9f7fd65742448dcb279f61823456"
format_stamp: "Formatted at 2026-09-29 15:49:33 on dist-test-slave-7491"
server_key: "0e3c369249dcb782c52483308f675b9e"
server_key_iv: "2636ea00b91de0b344ce6bff460bc9f4"
server_key_version: "encryptionkey@0"
I20260929 15:49:33.158011 17670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:33.176390 17670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:33.181317 17670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:33.181640 17670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:33.182772 17670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:33.184367 17670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:33.184451 17670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:33.184576 17670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:33.184648 17670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:33.227819 17670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46735
I20260929 15:49:33.227856 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46735 every 8 connection(s)
I20260929 15:49:33.229292 17670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:33.237167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17670
I20260929 15:49:33.237411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:49:33.238101 388 external_mini_cluster.cc:1428] Setting key 24161cb863f69da8ef0ea91aa54d71b4
Sep 29 15:49:33 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:33.257093 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:33.232827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55443 (local address 127.0.97.62:38995)
0929 15:49:33.233343 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:33.233348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:33.234410 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 15:49:33.236574 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:33.236579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:33.236684 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:33.236809 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:33.238733 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.239779 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.242368 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.242830 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.242993 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:33.247763 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:33.247770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:33.247773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:33.247830 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:33.252019 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:33.253485 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:33.253494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:33.253501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:33.253730 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:33.254389 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:33.254397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:33.254404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:33.254846 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:33.255024 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:33.256250 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0929 15:49:33.256572 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":289,"threads_started":1}
I20260929 15:49:33.257910 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:33.233370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38995 (local address 127.0.97.1:55443)
0929 15:49:33.234210 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:33.234255 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:33.236212 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:33.237097 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:33.237107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:33.237825 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:33.238517 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:33.238539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.240225 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.240235 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:33.242108 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:33.242137 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.243263 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.243274 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:33.243557 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:33.244399 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:33.244431 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:33.247324 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:33.252448 (+ 5124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:33.252462 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:33.252485 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:33.253228 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:33.253931 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:33.253935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:33.253939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:33.254205 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:33.255089 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:33.255099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:33.255689 (+ 590us) client_negotiation.cc:769] Sending connection context
0929 15:49:33.256036 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 15:49:33.256655 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":343,"threads_started":1}
I20260929 15:49:33.260051 17798 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38995
I20260929 15:49:33.260502 17798 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:33.261721 17798 heartbeater.cc:511] Master 127.0.97.62:38995 requested a full tablet report, sending...
I20260929 15:49:33.264816 17609 ts_manager.cc:194] Registered new tserver with Master: 5e7c9f7fd65742448dcb279f61823456 (127.0.97.1:46735)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 15:49:33.267135 17609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:49:33.301250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38995
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:33.678308 17804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:33.678632 17804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:33.678731 17804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:33.687574 17804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:33.687783 17804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:33.687925 17804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:33.699756 17804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:33.701202 17804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:33.702536 17804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:33.717172 17809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:33.717298 17812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:33.717398 17810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:33.718510 17804 server_base.cc:1034] running on GCE node
I20260929 15:49:33.719305 17804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:33.720474 17804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:33.721731 17804 hybrid_clock.cc:648] HybridClock initialized: now 1790696973721665 us; error 71 us; skew 500 ppm
I20260929 15:49:33.722090 17804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:33 dist-test-slave-7491 krb5kdc[17564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:33.726253 17804 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:49:33.728072 17804 webserver.cc:460] Webserver started at http://127.0.97.2:43269/ using document root <none> and password file <none>
I20260929 15:49:33.728940 17804 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:33.729053 17804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:33.729318 17804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:33.732650 17804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "c58a4ae0411f4870b01a179d4273dd86"
format_stamp: "Formatted at 2026-09-29 15:49:33 on dist-test-slave-7491"
server_key: "d09fb29aa417c285a110543ed5e58a37"
server_key_iv: "1c665fa33156068438daaf304d1f59b3"
server_key_version: "encryptionkey@0"
I20260929 15:49:33.733469 17804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c58a4ae0411f4870b01a179d4273dd86"
format_stamp: "Formatted at 2026-09-29 15:49:33 on dist-test-slave-7491"
server_key: "d09fb29aa417c285a110543ed5e58a37"
server_key_iv: "1c665fa33156068438daaf304d1f59b3"
server_key_version: "encryptionkey@0"
I20260929 15:49:33.739554 17804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:49:33.743682 17819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:33.745407 17804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 15:49:33.745584 17804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "c58a4ae0411f4870b01a179d4273dd86"
format_stamp: "Formatted at 2026-09-29 15:49:33 on dist-test-slave-7491"
server_key: "d09fb29aa417c285a110543ed5e58a37"
server_key_iv: "1c665fa33156068438daaf304d1f59b3"
server_key_version: "encryptionkey@0"
I20260929 15:49:33.745792 17804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:33.759121 17804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:33.764571 17804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:33.764837 17804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:33.765784 17804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:33.767396 17804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:33.767475 17804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:33.767616 17804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:33.767675 17804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:33.813167 17804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39843
I20260929 15:49:33.813335 17931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39843 every 8 connection(s)
I20260929 15:49:33.814579 17804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:33.822068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17804
I20260929 15:49:33.822268 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:49:33.822790 388 external_mini_cluster.cc:1428] Setting key fab598b08e3de8af8b3a7e14ffcfa01d
Sep 29 15:49:33 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:33.841938 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:33.817547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52831 (local address 127.0.97.62:38995)
0929 15:49:33.818074 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:33.818079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:33.819167 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 15:49:33.821197 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:33.821203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:33.821300 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:33.821416 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:33.823722 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.824608 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.827452 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.827949 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.828071 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:33.834445 (+ 6374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:33.834452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:33.834455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:33.834498 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:33.837750 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:33.838691 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:33.838700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:33.838706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:33.838869 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:33.839381 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:33.839388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:33.839395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:33.840132 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:33.840411 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:33.841123 (+ 712us) server_negotiation.cc:299] Negotiation successful
0929 15:49:33.841412 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":297,"threads_started":1}
I20260929 15:49:33.842613 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:33.818150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38995 (local address 127.0.97.2:52831)
0929 15:49:33.818964 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:33.819013 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:33.820874 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:33.821743 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:33.821758 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:33.822606 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:33.823514 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:33.823540 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.824779 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.824783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:33.827114 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:33.827148 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:33.828527 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:33.828541 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:33.829049 (+ 508us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:33.830453 (+ 1404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:33.830501 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:33.834045 (+ 3544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:33.837974 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:33.837983 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:33.838019 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:33.838539 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:33.839021 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:33.839029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:33.839032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:33.839259 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:33.840340 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:33.840347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:33.840730 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 15:49:33.840988 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 15:49:33.841437 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260929 15:49:33.844375 17932 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38995
I20260929 15:49:33.844754 17932 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:33.845664 17932 heartbeater.cc:511] Master 127.0.97.62:38995 requested a full tablet report, sending...
I20260929 15:49:33.847462 17609 ts_manager.cc:194] Registered new tserver with Master: c58a4ae0411f4870b01a179d4273dd86 (127.0.97.2:39843)
I20260929 15:49:33.848752 17609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52831
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:49:33.891172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38995
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:34.270120 17938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:34.270439 17938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:34.270617 17938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:49:34.270874 17798 heartbeater.cc:503] Master 127.0.97.62:38995 was elected leader, sending a full tablet report...
W20260929 15:49:34.278874 17938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:34.279115 17938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:34.279359 17938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:34.289870 17938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:34.291257 17938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:34.292691 17938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:34.308314 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:34.308456 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:34.308632 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:34.310426 17938 server_base.cc:1034] running on GCE node
I20260929 15:49:34.311216 17938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:34.312796 17938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:34.314124 17938 hybrid_clock.cc:648] HybridClock initialized: now 1790696974314025 us; error 67 us; skew 500 ppm
I20260929 15:49:34.314612 17938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:34 dist-test-slave-7491 krb5kdc[17564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:34.318336 17938 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:49:34.320652 17938 webserver.cc:460] Webserver started at http://127.0.97.3:41069/ using document root <none> and password file <none>
I20260929 15:49:34.321446 17938 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:34.321559 17938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:34.322080 17938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:34.325579 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "73a34843e0294077a36511d9f4450aee"
format_stamp: "Formatted at 2026-09-29 15:49:34 on dist-test-slave-7491"
server_key: "bbbb3d699db7fa203c76302809d47426"
server_key_iv: "57f9e1f7099afaff67fda11ad6b9bced"
server_key_version: "encryptionkey@0"
I20260929 15:49:34.326459 17938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "73a34843e0294077a36511d9f4450aee"
format_stamp: "Formatted at 2026-09-29 15:49:34 on dist-test-slave-7491"
server_key: "bbbb3d699db7fa203c76302809d47426"
server_key_iv: "57f9e1f7099afaff67fda11ad6b9bced"
server_key_version: "encryptionkey@0"
I20260929 15:49:34.332785 17938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:49:34.337212 17953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:34.339479 17938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:49:34.339767 17938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "73a34843e0294077a36511d9f4450aee"
format_stamp: "Formatted at 2026-09-29 15:49:34 on dist-test-slave-7491"
server_key: "bbbb3d699db7fa203c76302809d47426"
server_key_iv: "57f9e1f7099afaff67fda11ad6b9bced"
server_key_version: "encryptionkey@0"
I20260929 15:49:34.340042 17938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:34.372323 17938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:34.377737 17938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:34.378093 17938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:34.379208 17938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:34.380844 17938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:34.380923 17938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:34.381058 17938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:34.381114 17938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:34.419413 18065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35787 every 8 connection(s)
I20260929 15:49:34.419450 17938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35787
I20260929 15:49:34.421221 17938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:34.430775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 17938
I20260929 15:49:34.430970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:49:34.431555 388 external_mini_cluster.cc:1428] Setting key 91911743b79dd00a165c1a0223fe5e0c
Sep 29 15:49:34 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:34.445344 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.424065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49583 (local address 127.0.97.62:38995)
0929 15:49:34.424620 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.424625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.425586 (+ 961us) server_negotiation.cc:407] Connection header received
0929 15:49:34.427836 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.427842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.427935 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.428054 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:34.429686 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.430727 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.433522 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.433965 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.434275 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.438386 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:34.438392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:34.438396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:34.438435 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:34.441652 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:34.442701 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:34.442706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:34.442708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:34.442838 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:34.443308 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:34.443313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:34.443321 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:34.443616 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:34.443790 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.444612 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.444878 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":329,"threads_started":1}
I20260929 15:49:34.446631 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.424586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38995 (local address 127.0.97.3:49583)
0929 15:49:34.425400 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:34.425456 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:34.427485 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:34.428344 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:34.428354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:34.428904 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:34.429501 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.429519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.430899 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.430904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.433330 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.433347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.434367 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.434377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.434663 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.435524 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:34.435556 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:34.438095 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:34.442013 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:34.442022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:34.442039 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:34.442545 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:34.442958 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:34.442963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:34.442971 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:34.443180 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:34.443753 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:34.443762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:34.444145 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 15:49:34.444489 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 15:49:34.445189 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":319,"threads_started":1}
I20260929 15:49:34.448294 18066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38995
I20260929 15:49:34.448623 18066 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:34.449394 18066 heartbeater.cc:511] Master 127.0.97.62:38995 requested a full tablet report, sending...
I20260929 15:49:34.451273 17609 ts_manager.cc:194] Registered new tserver with Master: 73a34843e0294077a36511d9f4450aee (127.0.97.3:35787)
I20260929 15:49:34.452401 17609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49583
I20260929 15:49:34.460685 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:49:34.481406 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.465854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.0.97.62:38995)
0929 15:49:34.466156 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.466162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.466474 (+ 312us) server_negotiation.cc:407] Connection header received
0929 15:49:34.466711 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.466716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.466786 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.466894 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:34.468436 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.469414 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.470924 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.471364 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.471509 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.475155 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:34.475163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:34.475167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:34.475308 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:34.478500 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:34.479431 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:34.479435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:34.479438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:34.479599 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:34.480015 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:34.480018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:34.480020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:34.480447 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:34.480636 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.480965 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.481166 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:49:34.485965 17609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:49:34.495697 17609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:49:34.541014 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.511511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46735 (local address 127.0.0.1:50812)
0929 15:49:34.514184 (+ 2673us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:34.526583 (+ 12399us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:34.526869 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:34.530334 (+ 3465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:34.530343 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:34.530378 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:34.531005 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.531020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.533910 (+ 2890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.533916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.537393 (+ 3477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.537411 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.539749 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.539759 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.540005 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.540058 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 15:49:34.540222 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 15:49:34.540485 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2192,"mutex_wait_us":238,"thread_start_us":2029,"threads_started":1}
I20260929 15:49:34.541894 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.515465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.0.97.3:35787)
0929 15:49:34.528517 (+ 13052us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.528527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.528884 (+ 357us) server_negotiation.cc:407] Connection header received
0929 15:49:34.529767 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.529774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.529952 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.530250 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:34.532053 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.533710 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.538324 (+ 4614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.539845 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.539960 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.540273 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.541288 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.541512 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12842,"thread_start_us":12629,"threads_started":1}
I20260929 15:49:34.542608 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.511782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.0.97.1:46735)
0929 15:49:34.529447 (+ 17665us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.529457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.529486 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:49:34.529600 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.529612 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.529921 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.531392 (+ 1471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:34.531982 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.533737 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.537636 (+ 3899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.539445 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.541691 (+ 2246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.541978 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.542082 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.542282 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17074,"thread_start_us":17103,"threads_started":1}
I20260929 15:49:34.543695 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.511871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39843 (local address 127.0.0.1:50290)
0929 15:49:34.527687 (+ 15816us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:34.527731 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:34.527888 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:34.534207 (+ 6319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:34.534215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:34.534248 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:34.534876 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.534890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.537390 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.537396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.539671 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.539687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.542826 (+ 3139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.542834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.543023 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.543058 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:49:34.543136 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 15:49:34.543277 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15627,"mutex_wait_us":4,"thread_start_us":16566,"threads_started":1}
I20260929 15:49:34.544785 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.516246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35787 (local address 127.0.0.1:40180)
0929 15:49:34.528758 (+ 12512us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:34.528781 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:34.529539 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:34.530719 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:34.530724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:34.530756 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:34.531759 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.531778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.533915 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.533918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.538079 (+ 4161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.538096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.540473 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.540479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.540693 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.540729 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:49:34.540830 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:49:34.541028 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12380,"mutex_wait_us":14,"thread_start_us":11055,"threads_started":1}
I20260929 15:49:34.545285 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.512423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.0.97.2:39843)
0929 15:49:34.533327 (+ 20904us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.533337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.533367 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:49:34.533472 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.533480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.533750 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.534477 (+ 727us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:34.535221 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.537204 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.539960 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.542554 (+ 2594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.544410 (+ 1856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.544638 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.544708 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.544971 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20715,"thread_start_us":21476,"threads_started":1}
I20260929 15:49:34.546960 17733 tablet_service.cc:1467] Processing CreateTablet for tablet 649302a9b6b5438ab6c37d7b31971982 (DEFAULT_TABLE table=test-table [id=858d42c454ce42518a880130f9630e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:34.549263 17733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649302a9b6b5438ab6c37d7b31971982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:34.549511 18001 tablet_service.cc:1467] Processing CreateTablet for tablet 649302a9b6b5438ab6c37d7b31971982 (DEFAULT_TABLE table=test-table [id=858d42c454ce42518a880130f9630e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:34.549553 17867 tablet_service.cc:1467] Processing CreateTablet for tablet 649302a9b6b5438ab6c37d7b31971982 (DEFAULT_TABLE table=test-table [id=858d42c454ce42518a880130f9630e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:34.550889 18001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649302a9b6b5438ab6c37d7b31971982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:34.550932 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649302a9b6b5438ab6c37d7b31971982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:34.568837 18087 tablet_bootstrap.cc:492] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Bootstrap starting.
I20260929 15:49:34.573246 18085 tablet_bootstrap.cc:492] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Bootstrap starting.
I20260929 15:49:34.575155 18086 tablet_bootstrap.cc:492] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Bootstrap starting.
I20260929 15:49:34.575501 18087 tablet_bootstrap.cc:654] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:34.576807 18087 log.cc:824] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:34.578500 18085 tablet_bootstrap.cc:654] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:34.579727 18086 tablet_bootstrap.cc:654] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:34.579941 18085 log.cc:824] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:34.580034 18087 tablet_bootstrap.cc:492] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: No bootstrap required, opened a new log
I20260929 15:49:34.580381 18087 ts_tablet_manager.cc:1397] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 15:49:34.581151 18086 log.cc:824] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:34.583865 18085 tablet_bootstrap.cc:492] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: No bootstrap required, opened a new log
I20260929 15:49:34.584409 18086 tablet_bootstrap.cc:492] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: No bootstrap required, opened a new log
I20260929 15:49:34.584498 18085 ts_tablet_manager.cc:1397] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 15:49:34.584736 18086 ts_tablet_manager.cc:1397] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 15:49:34.592895 18087 raft_consensus.cc:348] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.593458 18087 raft_consensus.cc:374] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:34.593561 18087 raft_consensus.cc:729] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7c9f7fd65742448dcb279f61823456, State: Initialized, Role: FOLLOWER
I20260929 15:49:34.594323 18087 consensus_queue.cc:260] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.594465 18085 raft_consensus.cc:348] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.594921 18085 raft_consensus.cc:374] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:34.595041 18085 raft_consensus.cc:729] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a34843e0294077a36511d9f4450aee, State: Initialized, Role: FOLLOWER
I20260929 15:49:34.596318 18085 consensus_queue.cc:260] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.597047 18087 ts_tablet_manager.cc:1428] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 15:49:34.597049 18086 raft_consensus.cc:348] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.597680 18086 raft_consensus.cc:374] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:34.597795 18086 raft_consensus.cc:729] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58a4ae0411f4870b01a179d4273dd86, State: Initialized, Role: FOLLOWER
I20260929 15:49:34.598613 18086 consensus_queue.cc:260] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.598953 18066 heartbeater.cc:503] Master 127.0.97.62:38995 was elected leader, sending a full tablet report...
I20260929 15:49:34.601086 18085 ts_tablet_manager.cc:1428] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260929 15:49:34.602636 17932 heartbeater.cc:503] Master 127.0.97.62:38995 was elected leader, sending a full tablet report...
I20260929 15:49:34.603998 18086 ts_tablet_manager.cc:1428] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
W20260929 15:49:34.673115 18067 tablet.cc:2369] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:49:34.732637 17799 tablet.cc:2369] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:34.778172 18093 raft_consensus.cc:482] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:34.778484 18093 raft_consensus.cc:504] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.780159 18093 leader_election.cc:288] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e7c9f7fd65742448dcb279f61823456 (127.0.97.1:46735), 73a34843e0294077a36511d9f4450aee (127.0.97.3:35787)
I20260929 15:49:34.787418 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.780813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34953 (local address 127.0.97.1:46735)
0929 15:49:34.781002 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.781006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.781593 (+ 587us) server_negotiation.cc:407] Connection header received
0929 15:49:34.781926 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.781932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.782014 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.782088 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:34.782778 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.783512 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.785551 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.786372 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.786549 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.786670 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.786997 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.787144 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 15:49:34.787487 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.781870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53775 (local address 127.0.97.3:35787)
0929 15:49:34.782083 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.782089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.782289 (+ 200us) server_negotiation.cc:407] Connection header received
0929 15:49:34.782499 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.782505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.782574 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.782673 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:34.783187 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.783889 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.785716 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.786516 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.786671 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.786783 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.787082 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.787222 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:49:34.787621 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.781546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35787 (local address 127.0.97.2:53775)
0929 15:49:34.782185 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:34.782199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:34.782363 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:34.782699 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:34.782704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:34.782726 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:34.783038 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.783045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.784036 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.784041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.785556 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.785568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.786749 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.786756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.786930 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.786954 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:49:34.787055 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:49:34.787191 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":22,"thread_start_us":311,"threads_started":1}
I20260929 15:49:34.788092 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.780674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46735 (local address 127.0.97.2:34953)
0929 15:49:34.781461 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:34.781481 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:34.781679 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:34.782272 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:34.782275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:34.782303 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:34.782622 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.782630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.783658 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.783665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.785368 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:34.785383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.786639 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.786646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:34.786818 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.786847 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:49:34.786967 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 15:49:34.787123 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":29,"thread_start_us":427,"threads_started":1}
I20260929 15:49:34.789177 18021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649302a9b6b5438ab6c37d7b31971982" candidate_uuid: "c58a4ae0411f4870b01a179d4273dd86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a34843e0294077a36511d9f4450aee" is_pre_election: true
I20260929 15:49:34.789387 17753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649302a9b6b5438ab6c37d7b31971982" candidate_uuid: "c58a4ae0411f4870b01a179d4273dd86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7c9f7fd65742448dcb279f61823456" is_pre_election: true
I20260929 15:49:34.789737 18021 raft_consensus.cc:2454] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c58a4ae0411f4870b01a179d4273dd86 in term 0.
I20260929 15:49:34.790014 17753 raft_consensus.cc:2454] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c58a4ae0411f4870b01a179d4273dd86 in term 0.
I20260929 15:49:34.790699 17823 leader_election.cc:302] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73a34843e0294077a36511d9f4450aee, c58a4ae0411f4870b01a179d4273dd86; no voters:
I20260929 15:49:34.791119 18093 raft_consensus.cc:2781] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:49:34.791280 18093 raft_consensus.cc:482] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:49:34.791397 18093 raft_consensus.cc:3037] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:34.794028 18093 raft_consensus.cc:504] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.794934 18093 leader_election.cc:288] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [CANDIDATE]: Term 1 election: Requested vote from peers 5e7c9f7fd65742448dcb279f61823456 (127.0.97.1:46735), 73a34843e0294077a36511d9f4450aee (127.0.97.3:35787)
I20260929 15:49:34.795655 17753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649302a9b6b5438ab6c37d7b31971982" candidate_uuid: "c58a4ae0411f4870b01a179d4273dd86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7c9f7fd65742448dcb279f61823456"
I20260929 15:49:34.795912 17753 raft_consensus.cc:3037] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:34.795987 18021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649302a9b6b5438ab6c37d7b31971982" candidate_uuid: "c58a4ae0411f4870b01a179d4273dd86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73a34843e0294077a36511d9f4450aee"
I20260929 15:49:34.796341 18021 raft_consensus.cc:3037] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:34.799813 17753 raft_consensus.cc:2454] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c58a4ae0411f4870b01a179d4273dd86 in term 1.
I20260929 15:49:34.799813 18021 raft_consensus.cc:2454] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c58a4ae0411f4870b01a179d4273dd86 in term 1.
I20260929 15:49:34.800391 17823 leader_election.cc:302] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7c9f7fd65742448dcb279f61823456, c58a4ae0411f4870b01a179d4273dd86; no voters:
I20260929 15:49:34.800760 18093 raft_consensus.cc:2781] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:34.801272 18093 raft_consensus.cc:686] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 1 LEADER]: Becoming Leader. State: Replica: c58a4ae0411f4870b01a179d4273dd86, State: Running, Role: LEADER
I20260929 15:49:34.801782 18093 consensus_queue.cc:237] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } }
I20260929 15:49:34.806726 17608 catalog_manager.cc:5495] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 reported cstate change: term changed from 0 to 1, leader changed from <none> to c58a4ae0411f4870b01a179d4273dd86 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c58a4ae0411f4870b01a179d4273dd86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c58a4ae0411f4870b01a179d4273dd86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 } health_report { overall_health: UNKNOWN } } }
W20260929 15:49:34.817162 17933 tablet.cc:2369] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:34.833386 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.826723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.0.97.2:39843)
0929 15:49:34.826949 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.826956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.827056 (+ 100us) server_negotiation.cc:407] Connection header received
0929 15:49:34.827298 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.827303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.827377 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.827674 (+ 297us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:49:34.828233 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.829026 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.830498 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.830939 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.831066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.831611 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:49:34.832456 (+ 845us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:49:34.832860 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.832982 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.833075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:49:34.843114 17753 raft_consensus.cc:1261] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 1 FOLLOWER]: Refusing update from remote peer c58a4ae0411f4870b01a179d4273dd86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:34.843114 18021 raft_consensus.cc:1261] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 1 FOLLOWER]: Refusing update from remote peer c58a4ae0411f4870b01a179d4273dd86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:34.844457 18093 consensus_queue.cc:1035] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7c9f7fd65742448dcb279f61823456" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:34.844799 18096 consensus_queue.cc:1035] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73a34843e0294077a36511d9f4450aee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:34.858179 18099 mvcc.cc:205] Tried to move back new op lower bound from 7334694808944984064 to 7334694808792231936. Current Snapshot: MvccSnapshot[applied={T|T < 7334694808944984064}]
I20260929 15:49:34.864995 18101 mvcc.cc:205] Tried to move back new op lower bound from 7334694808944984064 to 7334694808792231936. Current Snapshot: MvccSnapshot[applied={T|T < 7334694808944984064 or (T in {7334694808944984064})}]
I20260929 15:49:34.876981 17608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:34.877301 17608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:34.881080 17608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f56555fdbc44957802a225c6c15079e: Sending DeleteTablet for 3 replicas of tablet 649302a9b6b5438ab6c37d7b31971982
I20260929 15:49:34.882442 17867 tablet_service.cc:1514] Processing DeleteTablet for tablet 649302a9b6b5438ab6c37d7b31971982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50290
I20260929 15:49:34.882790 17733 tablet_service.cc:1514] Processing DeleteTablet for tablet 649302a9b6b5438ab6c37d7b31971982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50812
I20260929 15:49:34.882921 18001 tablet_service.cc:1514] Processing DeleteTablet for tablet 649302a9b6b5438ab6c37d7b31971982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40180
I20260929 15:49:34.884233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17670
I20260929 15:49:34.894073 18111 tablet_replica.cc:330] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: stopping tablet replica
I20260929 15:49:34.894407 18109 tablet_replica.cc:330] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: stopping tablet replica
I20260929 15:49:34.894855 18109 raft_consensus.cc:2229] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:49:34.894843 18111 raft_consensus.cc:2229] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:34.895512 18111 raft_consensus.cc:2258] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:34.895707 18109 raft_consensus.cc:2258] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:34.898105 18110 tablet_replica.cc:330] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: stopping tablet replica
I20260929 15:49:34.898643 18110 raft_consensus.cc:2229] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:34.899116 18111 ts_tablet_manager.cc:1905] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:34.899436 18110 raft_consensus.cc:2258] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:34.901674 18109 ts_tablet_manager.cc:1905] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:34.910210 18110 ts_tablet_manager.cc:1905] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:34.910691 18109 ts_tablet_manager.cc:1918] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:34.910885 18109 log.cc:1196] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/649302a9b6b5438ab6c37d7b31971982
I20260929 15:49:34.911825 18109 ts_tablet_manager.cc:1939] T 649302a9b6b5438ab6c37d7b31971982 P c58a4ae0411f4870b01a179d4273dd86: Deleting consensus metadata
Sep 29 15:49:34 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:49:34.913822 18111 ts_tablet_manager.cc:1918] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:34.914057 18111 log.cc:1196] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/649302a9b6b5438ab6c37d7b31971982
I20260929 15:49:34.914130 17593 catalog_manager.cc:4841] TS c58a4ae0411f4870b01a179d4273dd86 (127.0.97.2:39843): tablet 649302a9b6b5438ab6c37d7b31971982 (table test-table [id=858d42c454ce42518a880130f9630e80]) successfully deleted
I20260929 15:49:34.914633 18111 ts_tablet_manager.cc:1939] T 649302a9b6b5438ab6c37d7b31971982 P 73a34843e0294077a36511d9f4450aee: Deleting consensus metadata
I20260929 15:49:34.916733 17596 catalog_manager.cc:4841] TS 73a34843e0294077a36511d9f4450aee (127.0.97.3:35787): tablet 649302a9b6b5438ab6c37d7b31971982 (table test-table [id=858d42c454ce42518a880130f9630e80]) successfully deleted
I20260929 15:49:34.919477 18110 ts_tablet_manager.cc:1918] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:34.919692 18110 log.cc:1196] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/649302a9b6b5438ab6c37d7b31971982
I20260929 15:49:34.920158 18110 ts_tablet_manager.cc:1939] T 649302a9b6b5438ab6c37d7b31971982 P 5e7c9f7fd65742448dcb279f61823456: Deleting consensus metadata
I20260929 15:49:34.922250 17596 catalog_manager.cc:4841] TS 5e7c9f7fd65742448dcb279f61823456 (127.0.97.1:46735): tablet 649302a9b6b5438ab6c37d7b31971982 (table test-table [id=858d42c454ce42518a880130f9630e80]) successfully deleted
I20260929 15:49:34.924890 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:34.885823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.0.97.1:46735)
0929 15:49:34.886035 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:34.886043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:34.901097 (+ 15054us) server_negotiation.cc:407] Connection header received
0929 15:49:34.901623 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:34.901631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:34.901725 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:34.901803 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:34.903719 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.905014 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.907196 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:34.908371 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:34.908487 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:34.915432 (+ 6945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:34.915454 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:34.915474 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:34.915526 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:34.921499 (+ 5973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:34.922520 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:34.922527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:34.922530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:34.922675 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:34.923199 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:34.923203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:34.923205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:34.923608 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:34.923836 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:34.924391 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 15:49:34.924582 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:49:34.925627 17793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:35.058771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17670
I20260929 15:49:35.078838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17804
Sep 29 15:49:35 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:49:35.101313 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:35.079819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.0.97.2:39843)
0929 15:49:35.080026 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:35.080033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:35.080060 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:49:35.080129 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:35.080134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:35.080239 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:35.080356 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:35.081577 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:35.082898 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:35.085122 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:35.085849 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:35.086027 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:35.091328 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:35.091352 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:35.091374 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:35.091443 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:35.097852 (+ 6409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:35.098810 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:35.098820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:35.098824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:35.099020 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:35.099454 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:35.099460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:35.099463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:35.099957 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:35.100340 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:35.100635 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 15:49:35.100895 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:49:35.102255 17927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:35.228945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17804
I20260929 15:49:35.249703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17938
Sep 29 15:49:35 dist-test-slave-7491 krb5kdc[17564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:49:35.268532 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:35.250647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.0.97.3:35787)
0929 15:49:35.250854 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:35.250862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:35.250934 (+ 72us) server_negotiation.cc:407] Connection header received
0929 15:49:35.251153 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:35.251158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:35.251252 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:35.251358 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:35.253129 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:35.253881 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:35.255424 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:35.256396 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:35.256612 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:35.262306 (+ 5694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:35.262325 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:35.262342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:35.262392 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:35.265318 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:35.266166 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:35.266174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:35.266176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:35.266336 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:35.266857 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:35.266861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:35.266863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:35.267259 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:35.267481 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:35.267961 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 15:49:35.268158 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:49:35.269207 18061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:35.364279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17938
I20260929 15:49:35.382530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 17577
I20260929 15:49:35.383812 17639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:35.502092 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 17577
2026-09-29T15:49:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4969 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T15:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:35Z Disabled control of system clock
I20260929 15:49:35.592469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46439
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:49:35.984982 18120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:35.985337 18120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:35.985430 18120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:35.993614 18120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:35.993749 18120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:35.993788 18120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:35.993841 18120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:35.993886 18120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:36.004604 18120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:36.005928 18120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:36.007844 18120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:36.022068 18126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:36.022167 18128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:36.022207 18125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:36.022783 18120 server_base.cc:1034] running on GCE node
I20260929 15:49:36.023680 18120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:36.025688 18120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:36.026947 18120 hybrid_clock.cc:648] HybridClock initialized: now 1790696976026883 us; error 70 us; skew 500 ppm
I20260929 15:49:36.027405 18120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:36.030433 18120 webserver.cc:460] Webserver started at http://127.0.97.62:37757/ using document root <none> and password file <none>
I20260929 15:49:36.031358 18120 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:36.031462 18120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:36.031816 18120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:36.035219 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "e199ab3852b241b8acbd2fbcd41d676f"
format_stamp: "Formatted at 2026-09-29 15:49:36 on dist-test-slave-7491"
server_key: "08240ff34dab1e8578b97c355fafdeb0"
server_key_iv: "52682e2f05c992d8d4b0b36c8d4cc325"
server_key_version: "encryptionkey@0"
I20260929 15:49:36.036118 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "e199ab3852b241b8acbd2fbcd41d676f"
format_stamp: "Formatted at 2026-09-29 15:49:36 on dist-test-slave-7491"
server_key: "08240ff34dab1e8578b97c355fafdeb0"
server_key_iv: "52682e2f05c992d8d4b0b36c8d4cc325"
server_key_version: "encryptionkey@0"
I20260929 15:49:36.042052 18120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:49:36.046136 18134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:36.047569 18120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:49:36.047793 18120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "e199ab3852b241b8acbd2fbcd41d676f"
format_stamp: "Formatted at 2026-09-29 15:49:36 on dist-test-slave-7491"
server_key: "08240ff34dab1e8578b97c355fafdeb0"
server_key_iv: "52682e2f05c992d8d4b0b36c8d4cc325"
server_key_version: "encryptionkey@0"
I20260929 15:49:36.048043 18120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:36.066294 18120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:36.067293 18120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:36.067626 18120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:36.089581 18120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46439
I20260929 15:49:36.089613 18185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46439 every 8 connection(s)
I20260929 15:49:36.091068 18120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:36.094205 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18120
I20260929 15:49:36.094403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:36.094937 388 external_mini_cluster.cc:1428] Setting key 220e25d9678134af5293561f7585f49a
I20260929 15:49:36.095904 18186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:36.110719 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:36.097293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.97.62:46439)
0929 15:49:36.098342 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:36.098354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:36.098395 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:49:36.100244 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:36.100277 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:36.100846 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:36.101254 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:36.101962 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:36.103369 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:36.105510 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:36.106267 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:36.106424 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:36.107952 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:36.107965 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:36.107997 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:36.108735 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:36.108824 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:36.108849 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:36.108959 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:36.109316 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 15:49:36.109717 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":414,"threads_started":1}
I20260929 15:49:36.112254 18186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f: Bootstrap starting.
I20260929 15:49:36.115899 18186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:36.117383 18186 log.cc:824] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:36.120916 18186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f: No bootstrap required, opened a new log
I20260929 15:49:36.130610 18186 raft_consensus.cc:348] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e199ab3852b241b8acbd2fbcd41d676f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260929 15:49:36.131306 18186 raft_consensus.cc:374] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:36.131443 18186 raft_consensus.cc:729] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e199ab3852b241b8acbd2fbcd41d676f, State: Initialized, Role: FOLLOWER
I20260929 15:49:36.132265 18186 consensus_queue.cc:260] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e199ab3852b241b8acbd2fbcd41d676f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260929 15:49:36.132539 18186 raft_consensus.cc:388] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:36.132691 18186 raft_consensus.cc:482] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:36.132918 18186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:36.136013 18186 raft_consensus.cc:504] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e199ab3852b241b8acbd2fbcd41d676f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260929 15:49:36.136785 18186 leader_election.cc:302] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e199ab3852b241b8acbd2fbcd41d676f; no voters:
I20260929 15:49:36.137437 18186 leader_election.cc:288] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:36.137651 18191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:36.138561 18191 raft_consensus.cc:686] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [term 1 LEADER]: Becoming Leader. State: Replica: e199ab3852b241b8acbd2fbcd41d676f, State: Running, Role: LEADER
I20260929 15:49:36.139154 18191 consensus_queue.cc:237] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e199ab3852b241b8acbd2fbcd41d676f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } }
I20260929 15:49:36.139760 18186 sys_catalog.cc:564] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:36.142076 18193 sys_catalog.cc:455] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e199ab3852b241b8acbd2fbcd41d676f. Latest consensus state: current_term: 1 leader_uuid: "e199ab3852b241b8acbd2fbcd41d676f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e199ab3852b241b8acbd2fbcd41d676f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } } }
I20260929 15:49:36.142416 18193 sys_catalog.cc:458] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:36.143370 18192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e199ab3852b241b8acbd2fbcd41d676f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e199ab3852b241b8acbd2fbcd41d676f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46439 } } }
I20260929 15:49:36.143736 18192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:36.145618 18197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:36.151427 18197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:36.162398 18197 catalog_manager.cc:1347] Generated new cluster ID: efdfe5c6d2c24045acb207819d63fe5e
I20260929 15:49:36.162563 18197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:36.197439 18197 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:36.199477 18197 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:36.209064 18197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f: Generated new TSK 0
I20260929 15:49:36.210032 18197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:49:36.217306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--builtin_ntp_servers=127.0.97.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:49:36.609737 18210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:36.610062 18210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:36.610208 18210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:36.619837 18210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:36.619979 18210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:36.620110 18210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:36.631304 18210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:36.633002 18210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:36.634917 18210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:36.650230 18216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:36.650195 18218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:36.650267 18215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:36.651252 18210 server_base.cc:1034] running on GCE node
I20260929 15:49:36.652026 18210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:36.653777 18210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:36.655340 18210 hybrid_clock.cc:648] HybridClock initialized: now 1790696976655268 us; error 72 us; skew 500 ppm
I20260929 15:49:36.655776 18210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:36.659119 18210 webserver.cc:460] Webserver started at http://127.0.97.1:46483/ using document root <none> and password file <none>
I20260929 15:49:36.660012 18210 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:36.660183 18210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:36.660599 18210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:36.663770 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "183786c4ee64416e88b6e060a1b2b5f0"
format_stamp: "Formatted at 2026-09-29 15:49:36 on dist-test-slave-7491"
server_key: "8d8c8a08ca15bf70374a3247dcded3fc"
server_key_iv: "7fde7784a0a573bc61ab05e2ceccc58a"
server_key_version: "encryptionkey@0"
I20260929 15:49:36.664707 18210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "183786c4ee64416e88b6e060a1b2b5f0"
format_stamp: "Formatted at 2026-09-29 15:49:36 on dist-test-slave-7491"
server_key: "8d8c8a08ca15bf70374a3247dcded3fc"
server_key_iv: "7fde7784a0a573bc61ab05e2ceccc58a"
server_key_version: "encryptionkey@0"
I20260929 15:49:36.670377 18210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:49:36.674863 18224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:36.676831 18210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:49:36.677003 18210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "183786c4ee64416e88b6e060a1b2b5f0"
format_stamp: "Formatted at 2026-09-29 15:49:36 on dist-test-slave-7491"
server_key: "8d8c8a08ca15bf70374a3247dcded3fc"
server_key_iv: "7fde7784a0a573bc61ab05e2ceccc58a"
server_key_version: "encryptionkey@0"
I20260929 15:49:36.677282 18210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:36.723799 18210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:36.724887 18210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:36.725221 18210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:36.726409 18210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:36.728272 18210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:36.728376 18210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:36.728490 18210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:36.728592 18210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:36.767879 18210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35725
I20260929 15:49:36.767930 18336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35725 every 8 connection(s)
I20260929 15:49:36.769291 18210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:36.771113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18210
I20260929 15:49:36.771291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:49:36.771823 388 external_mini_cluster.cc:1428] Setting key a7a6a022e03f955a1d60186df6f4f9d6
I20260929 15:49:36.775588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--builtin_ntp_servers=127.0.97.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:49:36.785712 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:36.772765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51479 (local address 127.0.97.62:46439)
0929 15:49:36.775156 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:36.775163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:36.775184 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:49:36.777170 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:36.777176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:36.777280 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:36.777407 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:36.778634 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:36.779379 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:36.781323 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:36.782125 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:36.782284 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:36.784746 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:36.784752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:36.784756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:36.784779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:36.784825 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:36.784838 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:36.784924 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:36.785332 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 15:49:36.785441 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2267,"thread_start_us":7471,"threads_started":1}
I20260929 15:49:36.787396 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:36.773386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.1:51479)
0929 15:49:36.774879 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:36.774945 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:36.776845 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:36.777685 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:36.777695 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:36.777807 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:36.778432 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:36.778449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:36.779623 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:36.779626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:36.781113 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:36.781129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:36.782531 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:36.782540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:36.783073 (+ 533us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:36.784222 (+ 1149us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:36.784228 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:36.784297 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:36.784438 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:36.784960 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:36.784976 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:36.784996 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 15:49:36.785199 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 15:49:36.785709 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":407,"threads_started":1}
I20260929 15:49:36.790226 18337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46439
I20260929 15:49:36.790778 18337 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:36.791961 18337 heartbeater.cc:511] Master 127.0.97.62:46439 requested a full tablet report, sending...
I20260929 15:49:36.795454 18151 ts_manager.cc:194] Registered new tserver with Master: 183786c4ee64416e88b6e060a1b2b5f0 (127.0.97.1:35725)
I20260929 15:49:36.797730 18151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51479
W20260929 15:49:37.136688 18341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:37.137090 18341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:37.137290 18341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:37.145015 18341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:37.145135 18341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:37.145303 18341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:37.155480 18341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:37.156891 18341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:37.158718 18341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:37.175235 18346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:37.175212 18347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:37.175278 18349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:37.176518 18341 server_base.cc:1034] running on GCE node
I20260929 15:49:37.177297 18341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:37.178483 18341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:37.179800 18341 hybrid_clock.cc:648] HybridClock initialized: now 1790696977179703 us; error 93 us; skew 500 ppm
I20260929 15:49:37.180327 18341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:37.183705 18341 webserver.cc:460] Webserver started at http://127.0.97.2:37877/ using document root <none> and password file <none>
I20260929 15:49:37.184856 18341 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:37.185034 18341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:37.185467 18341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:37.189702 18341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e5589058b3c4d06a820c58b904c0097"
format_stamp: "Formatted at 2026-09-29 15:49:37 on dist-test-slave-7491"
server_key: "da54f1b6ade526608c8a88c8a7bf09e2"
server_key_iv: "d9e897068c50f55d1c9eb180b360545f"
server_key_version: "encryptionkey@0"
I20260929 15:49:37.190855 18341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e5589058b3c4d06a820c58b904c0097"
format_stamp: "Formatted at 2026-09-29 15:49:37 on dist-test-slave-7491"
server_key: "da54f1b6ade526608c8a88c8a7bf09e2"
server_key_iv: "d9e897068c50f55d1c9eb180b360545f"
server_key_version: "encryptionkey@0"
I20260929 15:49:37.197363 18341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 15:49:37.201579 18355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:37.203714 18341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:49:37.204044 18341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "0e5589058b3c4d06a820c58b904c0097"
format_stamp: "Formatted at 2026-09-29 15:49:37 on dist-test-slave-7491"
server_key: "da54f1b6ade526608c8a88c8a7bf09e2"
server_key_iv: "d9e897068c50f55d1c9eb180b360545f"
server_key_version: "encryptionkey@0"
I20260929 15:49:37.204485 18341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:37.235584 18341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:37.236732 18341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:37.237044 18341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:37.238263 18341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:37.239965 18341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:37.240097 18341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:37.240255 18341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:37.240345 18341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:37.279392 18341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40023
I20260929 15:49:37.279489 18467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40023 every 8 connection(s)
I20260929 15:49:37.280853 18341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:37.287675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18341
I20260929 15:49:37.287892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:49:37.288420 388 external_mini_cluster.cc:1428] Setting key f07edb9c87cf0c4aa6a0a2e28d9523c8
I20260929 15:49:37.291815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--builtin_ntp_servers=127.0.97.20:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:49:37.297226 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.283853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51881 (local address 127.0.97.62:46439)
0929 15:49:37.284111 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:37.284116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:37.285683 (+ 1567us) server_negotiation.cc:407] Connection header received
0929 15:49:37.287740 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:37.287746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:37.287879 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:37.288020 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:37.289631 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.290705 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.293584 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.294359 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.294475 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.296397 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:37.296403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:37.296406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:37.296430 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:37.296466 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:37.296480 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:37.296553 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:37.296865 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:49:37.296961 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 15:49:37.298935 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.284443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.2:51881)
0929 15:49:37.285487 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:37.285535 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:37.287406 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:37.288528 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:37.288544 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:37.288665 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:37.289402 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.289437 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.290881 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.290887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.293343 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.293371 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.294633 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.294644 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.295026 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.295912 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:37.295918 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:37.295988 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:37.296122 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:37.296590 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:37.296603 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:37.296614 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:49:37.296831 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 15:49:37.297335 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":342,"threads_started":1}
I20260929 15:49:37.301141 18468 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46439
I20260929 15:49:37.301539 18468 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:37.302685 18468 heartbeater.cc:511] Master 127.0.97.62:46439 requested a full tablet report, sending...
I20260929 15:49:37.304905 18151 ts_manager.cc:194] Registered new tserver with Master: 0e5589058b3c4d06a820c58b904c0097 (127.0.97.2:40023)
I20260929 15:49:37.306198 18151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51881
W20260929 15:49:37.683482 18471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:37.683812 18471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:37.683902 18471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:37.691710 18471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:37.691915 18471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:37.692142 18471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:37.704602 18471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:37.706153 18471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:37.708122 18471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:37.725184 18476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:37.725116 18477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:37.725062 18479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:37.726193 18471 server_base.cc:1034] running on GCE node
I20260929 15:49:37.726933 18471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:37.728284 18471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:37.729601 18471 hybrid_clock.cc:648] HybridClock initialized: now 1790696977729509 us; error 62 us; skew 500 ppm
I20260929 15:49:37.730031 18471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:37.740557 18471 webserver.cc:460] Webserver started at http://127.0.97.3:36925/ using document root <none> and password file <none>
I20260929 15:49:37.741503 18471 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:37.741662 18471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:37.742192 18471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:37.745641 18471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "860b25cb994f43bd95e6bdc9869588ed"
format_stamp: "Formatted at 2026-09-29 15:49:37 on dist-test-slave-7491"
server_key: "323c5601a8cfceb111272e5afac14a20"
server_key_iv: "2f6d2c4f9ac945775e1026865de8ae8c"
server_key_version: "encryptionkey@0"
I20260929 15:49:37.746711 18471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "860b25cb994f43bd95e6bdc9869588ed"
format_stamp: "Formatted at 2026-09-29 15:49:37 on dist-test-slave-7491"
server_key: "323c5601a8cfceb111272e5afac14a20"
server_key_iv: "2f6d2c4f9ac945775e1026865de8ae8c"
server_key_version: "encryptionkey@0"
I20260929 15:49:37.754184 18471 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 15:49:37.759135 18485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:37.760886 18471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:37.761068 18471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "860b25cb994f43bd95e6bdc9869588ed"
format_stamp: "Formatted at 2026-09-29 15:49:37 on dist-test-slave-7491"
server_key: "323c5601a8cfceb111272e5afac14a20"
server_key_iv: "2f6d2c4f9ac945775e1026865de8ae8c"
server_key_version: "encryptionkey@0"
I20260929 15:49:37.761348 18471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:37.790815 18471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:37.791945 18471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:37.792423 18471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:37.794193 18471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:37.795971 18471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:37.796084 18471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:37.796257 18471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:37.796339 18471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:37.801468 18337 heartbeater.cc:503] Master 127.0.97.62:46439 was elected leader, sending a full tablet report...
I20260929 15:49:37.837385 18471 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39151
I20260929 15:49:37.837397 18597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39151 every 8 connection(s)
I20260929 15:49:37.838845 18471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:37.846047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18471
I20260929 15:49:37.846249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:49:37.846746 388 external_mini_cluster.cc:1428] Setting key 18167c2b82e5e49b3b0d0470d0eb600a
I20260929 15:49:37.855197 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.841928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45483 (local address 127.0.97.62:46439)
0929 15:49:37.842379 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:37.842384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:37.843603 (+ 1219us) server_negotiation.cc:407] Connection header received
0929 15:49:37.845697 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:37.845704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:37.845811 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:37.845908 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:37.848269 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.849461 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.851479 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.851956 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.852107 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.854154 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:37.854160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:37.854163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:37.854185 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:37.854233 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:37.854248 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:37.854361 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:37.854645 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 15:49:37.854778 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260929 15:49:37.856352 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.842450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46439 (local address 127.0.97.3:45483)
0929 15:49:37.843409 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:37.843453 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:37.845309 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:37.846737 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:37.846755 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:37.846902 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:37.848062 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.848095 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.849694 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.849700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.851292 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.851311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.852266 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.852276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.852630 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.853660 (+ 1030us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:37.853665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:37.853721 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:37.853842 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:37.854362 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:37.854376 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:37.854388 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:49:37.854612 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 15:49:37.855138 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
I20260929 15:49:37.858012 18598 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46439
I20260929 15:49:37.858378 18598 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:37.859220 18598 heartbeater.cc:511] Master 127.0.97.62:46439 requested a full tablet report, sending...
I20260929 15:49:37.860858 18151 ts_manager.cc:194] Registered new tserver with Master: 860b25cb994f43bd95e6bdc9869588ed (127.0.97.3:39151)
I20260929 15:49:37.861924 18151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:45483
I20260929 15:49:37.863355 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:49:37.876953 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.868336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.97.62:46439)
0929 15:49:37.868650 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:37.868658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:37.868987 (+ 329us) server_negotiation.cc:407] Connection header received
0929 15:49:37.869229 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:37.869235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:37.869314 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:37.869424 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:37.869935 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.870946 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.872819 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.873489 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.873808 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.875781 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:37.875787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:37.875790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:37.875813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:37.875849 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:37.875863 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:37.875964 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:37.876503 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 15:49:37.876623 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 15:49:37.882166 18151 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:49:37.890185 18151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:49:37.937211 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.905205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49392)
0929 15:49:37.920095 (+ 14890us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:37.920188 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:37.920479 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:37.924230 (+ 3751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:37.924243 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:37.924292 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:37.924804 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.924821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.931366 (+ 6545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.931374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.934102 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.934119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.936200 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.936207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.936567 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.936649 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 15:49:37.936768 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:49:37.936929 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14481,"mutex_wait_us":86,"thread_start_us":15510,"threads_started":1}
I20260929 15:49:37.938158 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.905352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.0.97.1:35725)
0929 15:49:37.923270 (+ 17918us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:37.923284 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:37.923328 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:49:37.923497 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:37.923509 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:37.923860 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:37.926778 (+ 2918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:37.927113 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.931189 (+ 4076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.934357 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.935955 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.936103 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.936426 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:37.937769 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0929 15:49:37.937910 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17719,"thread_start_us":17587,"threads_started":1}
I20260929 15:49:37.939664 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.905333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39151 (local address 127.0.0.1:48026)
0929 15:49:37.930390 (+ 25057us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:37.930420 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:37.930618 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:37.932240 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:37.932245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:37.932274 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:37.932678 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.932688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.934461 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.934466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.936566 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.936581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.938828 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.938835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.939028 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.939065 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:49:37.939152 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 15:49:37.939296 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24844,"mutex_wait_us":99,"thread_start_us":26313,"threads_started":1}
I20260929 15:49:37.940551 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.906077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.0.97.3:39151)
0929 15:49:37.931403 (+ 25326us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:37.931415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:37.931463 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:49:37.931608 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:37.931620 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:37.931927 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:37.932246 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:37.932938 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.934282 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.936800 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.938274 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.938409 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.938701 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:37.939949 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0929 15:49:37.940136 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25072,"thread_start_us":24876,"threads_started":1}
I20260929 15:49:37.941071 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.906683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.0.97.2:40023)
0929 15:49:37.929351 (+ 22668us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:37.929368 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:37.932121 (+ 2753us) server_negotiation.cc:407] Connection header received
0929 15:49:37.932384 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:37.932391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:37.932612 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:37.932807 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:37.933478 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.935082 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.937692 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.939100 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.939247 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.939452 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:37.940585 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0929 15:49:37.940774 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22233,"thread_start_us":22733,"threads_started":1}
I20260929 15:49:37.941318 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:37.905994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40023 (local address 127.0.0.1:40964)
0929 15:49:37.931946 (+ 25952us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:37.931974 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:37.932115 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:37.932819 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:37.932824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:37.932855 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:37.933291 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.933303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.935279 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.935286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.937456 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:37.937471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:37.939587 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:37.939599 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:37.939853 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:37.939896 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 15:49:37.939980 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:49:37.940117 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25813,"mutex_wait_us":1,"thread_start_us":25272,"threads_started":1}
I20260929 15:49:37.944491 18533 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea (DEFAULT_TABLE table=test-table [id=3aa4f15fb3e6473e8437f136d15f536e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:37.944566 18403 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea (DEFAULT_TABLE table=test-table [id=3aa4f15fb3e6473e8437f136d15f536e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:37.946025 18403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:37.946100 18533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:37.945457 18272 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea (DEFAULT_TABLE table=test-table [id=3aa4f15fb3e6473e8437f136d15f536e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:37.947242 18272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:37.960460 18617 tablet_bootstrap.cc:492] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Bootstrap starting.
I20260929 15:49:37.962822 18618 tablet_bootstrap.cc:492] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Bootstrap starting.
I20260929 15:49:37.963687 18619 tablet_bootstrap.cc:492] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Bootstrap starting.
I20260929 15:49:37.966405 18617 tablet_bootstrap.cc:654] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:37.967564 18618 tablet_bootstrap.cc:654] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:37.967624 18619 tablet_bootstrap.cc:654] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:37.967799 18617 log.cc:824] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:37.968883 18619 log.cc:824] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:37.968982 18618 log.cc:824] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:37.971788 18619 tablet_bootstrap.cc:492] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: No bootstrap required, opened a new log
I20260929 15:49:37.972045 18618 tablet_bootstrap.cc:492] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: No bootstrap required, opened a new log
I20260929 15:49:37.972146 18619 ts_tablet_manager.cc:1397] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:49:37.972393 18618 ts_tablet_manager.cc:1397] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 15:49:37.974977 18617 tablet_bootstrap.cc:492] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: No bootstrap required, opened a new log
I20260929 15:49:37.975265 18617 ts_tablet_manager.cc:1397] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260929 15:49:37.980316 18619 raft_consensus.cc:348] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:37.980754 18619 raft_consensus.cc:374] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:37.980911 18619 raft_consensus.cc:729] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183786c4ee64416e88b6e060a1b2b5f0, State: Initialized, Role: FOLLOWER
I20260929 15:49:37.981612 18619 consensus_queue.cc:260] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:37.983441 18619 ts_tablet_manager.cc:1428] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 15:49:37.983551 18618 raft_consensus.cc:348] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:37.984026 18618 raft_consensus.cc:374] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:37.984133 18618 raft_consensus.cc:729] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860b25cb994f43bd95e6bdc9869588ed, State: Initialized, Role: FOLLOWER
I20260929 15:49:37.984771 18618 consensus_queue.cc:260] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:37.985915 18598 heartbeater.cc:503] Master 127.0.97.62:46439 was elected leader, sending a full tablet report...
I20260929 15:49:37.986603 18618 ts_tablet_manager.cc:1428] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 15:49:37.987233 18617 raft_consensus.cc:348] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:37.987766 18617 raft_consensus.cc:374] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:37.987886 18617 raft_consensus.cc:729] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5589058b3c4d06a820c58b904c0097, State: Initialized, Role: FOLLOWER
I20260929 15:49:37.988628 18617 consensus_queue.cc:260] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:37.990509 18617 ts_tablet_manager.cc:1428] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 15:49:37.990631 18468 heartbeater.cc:503] Master 127.0.97.62:46439 was elected leader, sending a full tablet report...
W20260929 15:49:38.022521 18338 tablet.cc:2369] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:49:38.033403 18469 tablet.cc:2369] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:49:38.090912 18599 tablet.cc:2369] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:38.123591 18624 raft_consensus.cc:482] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:38.123934 18624 raft_consensus.cc:504] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:38.125842 18624 leader_election.cc:288] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 183786c4ee64416e88b6e060a1b2b5f0 (127.0.97.1:35725), 0e5589058b3c4d06a820c58b904c0097 (127.0.97.2:40023)
I20260929 15:49:38.132931 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.126389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40023 (local address 127.0.97.3:50967)
0929 15:49:38.126706 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.126734 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:38.126925 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:38.127381 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:38.127385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:38.127426 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:38.127654 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:38.127663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.128958 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.128962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:38.130914 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:38.130931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.132318 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.132322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:38.132452 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.132477 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 15:49:38.132578 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:49:38.132690 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1}
I20260929 15:49:38.133066 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.126431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50967 (local address 127.0.97.2:40023)
0929 15:49:38.126664 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:38.126671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:38.126912 (+ 241us) server_negotiation.cc:407] Connection header received
0929 15:49:38.127112 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:38.127119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:38.127222 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:38.127310 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:38.127846 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.128798 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.131107 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.132066 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.132252 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.132360 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:38.132638 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 15:49:38.132782 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:49:38.133965 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.126493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53781 (local address 127.0.97.1:35725)
0929 15:49:38.126831 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:38.126840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:38.127391 (+ 551us) server_negotiation.cc:407] Connection header received
0929 15:49:38.127660 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:38.127665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:38.127822 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:38.127961 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:38.128824 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.129560 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.131760 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.132947 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.133088 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.133192 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:38.133672 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 15:49:38.133797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260929 15:49:38.134054 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.126389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.97.3:53781)
0929 15:49:38.127257 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.127271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:38.127471 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:38.128083 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:38.128088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:38.128130 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:38.128640 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:38.128652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.129739 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.129745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:38.131552 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:38.131563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.133285 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.133291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:38.133444 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.133466 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:49:38.133552 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 15:49:38.133646 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":122,"spinlock_wait_cycles":41344,"thread_start_us":434,"threads_started":1}
I20260929 15:49:38.134438 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef96730b4ce44ec8dcf978a9fecf5ea" candidate_uuid: "860b25cb994f43bd95e6bdc9869588ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5589058b3c4d06a820c58b904c0097" is_pre_election: true
I20260929 15:49:38.135074 18423 raft_consensus.cc:2454] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860b25cb994f43bd95e6bdc9869588ed in term 0.
I20260929 15:49:38.135375 18292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef96730b4ce44ec8dcf978a9fecf5ea" candidate_uuid: "860b25cb994f43bd95e6bdc9869588ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183786c4ee64416e88b6e060a1b2b5f0" is_pre_election: true
I20260929 15:49:38.135893 18486 leader_election.cc:302] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5589058b3c4d06a820c58b904c0097, 860b25cb994f43bd95e6bdc9869588ed; no voters:
I20260929 15:49:38.135914 18292 raft_consensus.cc:2454] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860b25cb994f43bd95e6bdc9869588ed in term 0.
I20260929 15:49:38.136359 18624 raft_consensus.cc:2781] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:49:38.136547 18624 raft_consensus.cc:482] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:49:38.136691 18624 raft_consensus.cc:3037] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:38.138980 18624 raft_consensus.cc:504] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:38.139659 18624 leader_election.cc:288] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [CANDIDATE]: Term 1 election: Requested vote from peers 183786c4ee64416e88b6e060a1b2b5f0 (127.0.97.1:35725), 0e5589058b3c4d06a820c58b904c0097 (127.0.97.2:40023)
I20260929 15:49:38.140326 18292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef96730b4ce44ec8dcf978a9fecf5ea" candidate_uuid: "860b25cb994f43bd95e6bdc9869588ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183786c4ee64416e88b6e060a1b2b5f0"
I20260929 15:49:38.140578 18292 raft_consensus.cc:3037] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:38.140610 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef96730b4ce44ec8dcf978a9fecf5ea" candidate_uuid: "860b25cb994f43bd95e6bdc9869588ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5589058b3c4d06a820c58b904c0097"
I20260929 15:49:38.140893 18423 raft_consensus.cc:3037] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:38.143010 18292 raft_consensus.cc:2454] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860b25cb994f43bd95e6bdc9869588ed in term 1.
I20260929 15:49:38.143306 18423 raft_consensus.cc:2454] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860b25cb994f43bd95e6bdc9869588ed in term 1.
I20260929 15:49:38.143678 18488 leader_election.cc:302] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183786c4ee64416e88b6e060a1b2b5f0, 860b25cb994f43bd95e6bdc9869588ed; no voters:
I20260929 15:49:38.144073 18624 raft_consensus.cc:2781] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:38.144598 18624 raft_consensus.cc:686] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 1 LEADER]: Becoming Leader. State: Replica: 860b25cb994f43bd95e6bdc9869588ed, State: Running, Role: LEADER
I20260929 15:49:38.145095 18624 consensus_queue.cc:237] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } }
I20260929 15:49:38.150734 18150 catalog_manager.cc:5495] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 860b25cb994f43bd95e6bdc9869588ed (127.0.97.3). New cstate: current_term: 1 leader_uuid: "860b25cb994f43bd95e6bdc9869588ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "860b25cb994f43bd95e6bdc9869588ed" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 } health_report { overall_health: UNKNOWN } } }
I20260929 15:49:38.179754 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.173214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.0.97.3:39151)
0929 15:49:38.173419 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:38.173426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:38.173447 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:49:38.173592 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:38.173596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:38.173676 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:38.173830 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:49:38.174223 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.175107 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.177435 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.177893 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.178039 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.178591 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:49:38.179126 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:49:38.179298 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:38.179355 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 15:49:38.179471 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:49:38.189045 18292 raft_consensus.cc:1261] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 1 FOLLOWER]: Refusing update from remote peer 860b25cb994f43bd95e6bdc9869588ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:38.189124 18423 raft_consensus.cc:1261] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 1 FOLLOWER]: Refusing update from remote peer 860b25cb994f43bd95e6bdc9869588ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:38.190522 18627 consensus_queue.cc:1035] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "183786c4ee64416e88b6e060a1b2b5f0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:38.191231 18624 consensus_queue.cc:1035] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5589058b3c4d06a820c58b904c0097" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:38.234661 18151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:38.235002 18151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:38.239078 18151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e199ab3852b241b8acbd2fbcd41d676f: Sending DeleteTablet for 3 replicas of tablet 8ef96730b4ce44ec8dcf978a9fecf5ea
I20260929 15:49:38.240411 18403 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:38 UTC) from {username='slave'} at 127.0.0.1:40964
I20260929 15:49:38.240509 18533 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:38 UTC) from {username='slave'} at 127.0.0.1:48026
I20260929 15:49:38.241531 18640 tablet_replica.cc:330] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: stopping tablet replica
I20260929 15:49:38.241458 18272 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:38 UTC) from {username='slave'} at 127.0.0.1:49392
I20260929 15:49:38.242106 18640 raft_consensus.cc:2229] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:38.242709 18640 raft_consensus.cc:2258] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:38.243040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18210
I20260929 15:49:38.246295 18642 tablet_replica.cc:330] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: stopping tablet replica
I20260929 15:49:38.247000 18642 raft_consensus.cc:2229] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:38.248061 18642 raft_consensus.cc:2258] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:38.251497 18640 ts_tablet_manager.cc:1905] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:38.251879 18642 ts_tablet_manager.cc:1905] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:38.260474 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.244231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.0.97.1:35725)
0929 15:49:38.244416 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:38.244423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:38.249109 (+ 4686us) server_negotiation.cc:407] Connection header received
0929 15:49:38.249324 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:38.249330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:38.249450 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:38.249576 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:38.250275 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.251588 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.254778 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.256098 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.256264 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.258347 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:38.258362 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:38.258382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:38.259321 (+ 939us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:38.259390 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:38.259433 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:38.259542 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:38.259888 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 15:49:38.260021 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:49:38.261760 18642 ts_tablet_manager.cc:1918] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:38.261961 18642 log.cc:1196] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/8ef96730b4ce44ec8dcf978a9fecf5ea
I20260929 15:49:38.262271 18332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:38.263152 18641 tablet_replica.cc:330] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: stopping tablet replica
I20260929 15:49:38.263785 18641 raft_consensus.cc:2229] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:49:38.264948 18641 raft_consensus.cc:2258] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:38.267611 18641 ts_tablet_manager.cc:1905] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:38.269985 18640 ts_tablet_manager.cc:1918] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:38.270179 18640 log.cc:1196] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/8ef96730b4ce44ec8dcf978a9fecf5ea
I20260929 15:49:38.270725 18640 ts_tablet_manager.cc:1939] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 0e5589058b3c4d06a820c58b904c0097: Deleting consensus metadata
I20260929 15:49:38.272903 18135 catalog_manager.cc:4841] TS 0e5589058b3c4d06a820c58b904c0097 (127.0.97.2:40023): tablet 8ef96730b4ce44ec8dcf978a9fecf5ea (table test-table [id=3aa4f15fb3e6473e8437f136d15f536e]) successfully deleted
I20260929 15:49:38.277462 18641 ts_tablet_manager.cc:1918] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:38.277767 18641 log.cc:1196] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/8ef96730b4ce44ec8dcf978a9fecf5ea
I20260929 15:49:38.278378 18641 ts_tablet_manager.cc:1939] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 860b25cb994f43bd95e6bdc9869588ed: Deleting consensus metadata
I20260929 15:49:38.280694 18138 catalog_manager.cc:4841] TS 860b25cb994f43bd95e6bdc9869588ed (127.0.97.3:39151): tablet 8ef96730b4ce44ec8dcf978a9fecf5ea (table test-table [id=3aa4f15fb3e6473e8437f136d15f536e]) successfully deleted
I20260929 15:49:38.384688 18642 ts_tablet_manager.cc:1939] T 8ef96730b4ce44ec8dcf978a9fecf5ea P 183786c4ee64416e88b6e060a1b2b5f0: Deleting consensus metadata
I20260929 15:49:38.385459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18210
I20260929 15:49:38.413733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18341
W20260929 15:49:38.414574 18137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35725 (error 108)
I20260929 15:49:38.417232 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.416697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49418)
0929 15:49:38.416908 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.417038 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 15:49:38.417668 18137 catalog_manager.cc:4584] TS 183786c4ee64416e88b6e060a1b2b5f0 (127.0.97.1:35725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ef96730b4ce44ec8dcf978a9fecf5ea: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
I20260929 15:49:38.423632 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.415614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.0.97.2:40023)
0929 15:49:38.415878 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:38.415885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:38.415909 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:49:38.415981 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:38.416046 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:38.416349 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:38.416500 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:38.417004 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.417767 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.419226 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.419648 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.420052 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.421775 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:38.421789 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:38.421812 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:38.422677 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:38.422754 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:38.422782 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:38.422941 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:38.423103 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 15:49:38.423236 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 15:49:38.424582 18463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:38.472769 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.471758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49424)
0929 15:49:38.472297 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.472409 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330}
I20260929 15:49:38.475442 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.474921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49434)
0929 15:49:38.475149 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.475258 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 15:49:38.526252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18341
I20260929 15:49:38.536566 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.536033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49450)
0929 15:49:38.536306 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.536389 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 15:49:38.538033 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.537654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49460)
0929 15:49:38.537817 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.537911 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 15:49:38.553761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18471
I20260929 15:49:38.561421 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.554712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.0.97.3:39151)
0929 15:49:38.554938 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:38.554943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:38.554961 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:49:38.555016 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:38.555019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:38.555092 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:38.555222 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:38.555609 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.556465 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.557768 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:38.558213 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:38.558385 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:38.559635 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:38.559647 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:38.559661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:38.560569 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:38.560624 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:38.560645 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:38.560791 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:38.560936 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 15:49:38.561059 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:49:38.562247 18593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:38.618942 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.618390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49470)
0929 15:49:38.618649 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.618749 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 15:49:38.620139 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.619693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49480)
0929 15:49:38.619913 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.620024 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 15:49:38.695981 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18471
I20260929 15:49:38.714561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18120
I20260929 15:49:38.716281 18181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:38.844223 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:38.843377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35725 (local address 127.0.0.1:49492)
0929 15:49:38.843803 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:38.843940 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260929 15:49:38.844524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18120
2026-09-29T15:49:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3337 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:49:38 dist-test-slave-7491 krb5kdc[18651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:49:38 dist-test-slave-7491 krb5kdc[18651](info): set up 2 sockets
Sep 29 15:49:38 dist-test-slave-7491 krb5kdc[18651](info): commencing operation
krb5kdc: starting...
W20260929 15:49:40.016625 388 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.006s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:49:40 dist-test-slave-7491 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:49:40.299249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34623
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:40.680311 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:40.680631 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:40.680723 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:40.688292 18664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:40.688418 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:40.688468 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:40.688529 18664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:40.688575 18664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:40.698881 18664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:40.700228 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:40.701473 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:40.713821 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:40.713929 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:40.714504 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:40.715502 18664 server_base.cc:1034] running on GCE node
I20260929 15:49:40.716460 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:40.718063 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:40.719336 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790696980719289 us; error 42 us; skew 500 ppm
I20260929 15:49:40.719755 18664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:40 dist-test-slave-7491 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:40.723301 18664 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:49:40.725236 18664 webserver.cc:460] Webserver started at http://127.0.97.62:34703/ using document root <none> and password file <none>
I20260929 15:49:40.726012 18664 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:40.726188 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:40.726522 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:40.730190 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "3737a8797eba4bce9db4dc3752f86271"
format_stamp: "Formatted at 2026-09-29 15:49:40 on dist-test-slave-7491"
server_key: "ee765781ec9d78b4d87349b3ad54d651"
server_key_iv: "c531056914712a6f8d218cf9e0e65cdf"
server_key_version: "encryptionkey@0"
I20260929 15:49:40.731048 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "3737a8797eba4bce9db4dc3752f86271"
format_stamp: "Formatted at 2026-09-29 15:49:40 on dist-test-slave-7491"
server_key: "ee765781ec9d78b4d87349b3ad54d651"
server_key_iv: "c531056914712a6f8d218cf9e0e65cdf"
server_key_version: "encryptionkey@0"
I20260929 15:49:40.736931 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 15:49:40.741250 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:40.742678 18664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:49:40.742909 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "3737a8797eba4bce9db4dc3752f86271"
format_stamp: "Formatted at 2026-09-29 15:49:40 on dist-test-slave-7491"
server_key: "ee765781ec9d78b4d87349b3ad54d651"
server_key_iv: "c531056914712a6f8d218cf9e0e65cdf"
server_key_version: "encryptionkey@0"
I20260929 15:49:40.743150 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:40.762081 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:40.766571 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:40.766906 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:40.787356 18664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34623
I20260929 15:49:40.787386 18730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34623 every 8 connection(s)
I20260929 15:49:40.788960 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:40.789244 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18664
I20260929 15:49:40.789449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:40.789908 388 external_mini_cluster.cc:1428] Setting key c45c7dabc6b7529ef2596399877efc7b
I20260929 15:49:40.794088 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:49:40 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:40.808568 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271: Bootstrap starting.
I20260929 15:49:40.812440 18731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:40.813819 18731 log.cc:824] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:40.817121 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:40.791951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.0.97.62:34623)
0929 15:49:40.792831 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:40.792847 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:40.792898 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:49:40.795314 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:40.795344 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:40.795868 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:40.796536 (+ 668us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:40.798065 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:40.800274 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:40.801917 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:40.803007 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:40.803244 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:40.808346 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:40.808357 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:40.808375 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:40.808424 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:40.811887 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:40.813079 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:40.813088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:40.813094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:40.813283 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:40.813751 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:40.813754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:40.813756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:40.814481 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:40.814840 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:40.815560 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 15:49:40.815937 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":400,"threads_started":1}
I20260929 15:49:40.817943 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271: No bootstrap required, opened a new log
I20260929 15:49:40.826892 18731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3737a8797eba4bce9db4dc3752f86271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34623 } }
I20260929 15:49:40.827386 18731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:40.827510 18731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3737a8797eba4bce9db4dc3752f86271, State: Initialized, Role: FOLLOWER
I20260929 15:49:40.828225 18731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3737a8797eba4bce9db4dc3752f86271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34623 } }
I20260929 15:49:40.828509 18731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:40.828671 18731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:40.828869 18731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:40.831754 18731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3737a8797eba4bce9db4dc3752f86271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34623 } }
I20260929 15:49:40.832463 18731 leader_election.cc:302] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3737a8797eba4bce9db4dc3752f86271; no voters:
I20260929 15:49:40.833387 18731 leader_election.cc:288] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:40.833441 18736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:40.834173 18736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [term 1 LEADER]: Becoming Leader. State: Replica: 3737a8797eba4bce9db4dc3752f86271, State: Running, Role: LEADER
I20260929 15:49:40.834945 18736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3737a8797eba4bce9db4dc3752f86271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34623 } }
I20260929 15:49:40.836232 18731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:40.837841 18738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3737a8797eba4bce9db4dc3752f86271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3737a8797eba4bce9db4dc3752f86271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34623 } } }
I20260929 15:49:40.837785 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3737a8797eba4bce9db4dc3752f86271. Latest consensus state: current_term: 1 leader_uuid: "3737a8797eba4bce9db4dc3752f86271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3737a8797eba4bce9db4dc3752f86271" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34623 } } }
I20260929 15:49:40.840556 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:40.840862 18738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:40.849323 18743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:40.856045 18743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:40.866798 18743 catalog_manager.cc:1347] Generated new cluster ID: 3c45a3eafe5542bea007745ec28c42bd
I20260929 15:49:40.866963 18743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:40.900108 18743 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:40.901647 18743 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:40.911353 18743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271: Generated new TSK 0
I20260929 15:49:40.912385 18743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:49:40.989931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34623
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:41.350821 18757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:41.351133 18757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:41.351234 18757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:41.359228 18757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:41.359383 18757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:41.359505 18757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:41.370522 18757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:41.371847 18757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:41.373229 18757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:41.386684 18765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:41.386751 18763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:41.386972 18762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:41.387984 18757 server_base.cc:1034] running on GCE node
I20260929 15:49:41.388711 18757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:41.389736 18757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:41.390950 18757 hybrid_clock.cc:648] HybridClock initialized: now 1790696981390908 us; error 29 us; skew 500 ppm
I20260929 15:49:41.391321 18757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:41 dist-test-slave-7491 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:41.394692 18757 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:49:41.396399 18757 webserver.cc:460] Webserver started at http://127.0.97.1:45133/ using document root <none> and password file <none>
I20260929 15:49:41.397141 18757 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:41.397286 18757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:41.397617 18757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:41.400746 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "70231b6fa29f41cc9feb6fd538393219"
format_stamp: "Formatted at 2026-09-29 15:49:41 on dist-test-slave-7491"
server_key: "65c7bf22b14d187eb8d3fb669b6a9642"
server_key_iv: "488aed87a001d2cbcf786339f3896eb7"
server_key_version: "encryptionkey@0"
I20260929 15:49:41.401574 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "70231b6fa29f41cc9feb6fd538393219"
format_stamp: "Formatted at 2026-09-29 15:49:41 on dist-test-slave-7491"
server_key: "65c7bf22b14d187eb8d3fb669b6a9642"
server_key_iv: "488aed87a001d2cbcf786339f3896eb7"
server_key_version: "encryptionkey@0"
I20260929 15:49:41.406908 18757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:49:41.410691 18772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:41.412318 18757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:49:41.412503 18757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "70231b6fa29f41cc9feb6fd538393219"
format_stamp: "Formatted at 2026-09-29 15:49:41 on dist-test-slave-7491"
server_key: "65c7bf22b14d187eb8d3fb669b6a9642"
server_key_iv: "488aed87a001d2cbcf786339f3896eb7"
server_key_version: "encryptionkey@0"
I20260929 15:49:41.412724 18757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:41.424957 18757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:41.429780 18757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:41.430109 18757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:41.431114 18757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:41.432613 18757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:41.432685 18757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:41.432828 18757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:41.432880 18757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:41.467882 18757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39043
I20260929 15:49:41.467928 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39043 every 8 connection(s)
I20260929 15:49:41.469230 18757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:41.477237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18757
I20260929 15:49:41.477425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:49:41.477860 388 external_mini_cluster.cc:1428] Setting key 4fed95089b67325492f9d14cb140bc68
Sep 29 15:49:41 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:41.491511 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:41.471906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48637 (local address 127.0.97.62:34623)
0929 15:49:41.472406 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:41.472410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:41.473359 (+ 949us) server_negotiation.cc:407] Connection header received
0929 15:49:41.475327 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:41.475333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:41.475432 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:41.475557 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:41.477300 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:41.478187 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:41.480129 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:41.480584 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:41.480739 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:41.484935 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:41.484941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:41.484945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:41.484980 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:41.487713 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:41.488637 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:41.488645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:41.488648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:41.488815 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:41.489335 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:41.489344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:41.489350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:41.489766 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:41.489925 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:41.490753 (+ 828us) server_negotiation.cc:299] Negotiation successful
0929 15:49:41.491026 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":288,"threads_started":1}
I20260929 15:49:41.492385 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:41.472446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34623 (local address 127.0.97.1:48637)
0929 15:49:41.473184 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:41.473234 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:41.475004 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:41.475816 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:41.475825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:41.476489 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:41.477098 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:41.477126 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:41.478387 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:41.478392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:41.479978 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:41.479993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:41.480844 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:41.480853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:41.481134 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:41.481886 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:41.481911 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:41.484620 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:41.487894 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:41.487902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:41.487917 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:41.488479 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:41.488930 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:41.488934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:41.488936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:41.489209 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:41.489904 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:41.489911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:41.490345 (+ 434us) client_negotiation.cc:769] Sending connection context
0929 15:49:41.490589 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 15:49:41.491027 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":280,"threads_started":1}
I20260929 15:49:41.494071 18885 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34623
I20260929 15:49:41.494416 18885 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:41.495304 18885 heartbeater.cc:511] Master 127.0.97.62:34623 requested a full tablet report, sending...
I20260929 15:49:41.497692 18696 ts_manager.cc:194] Registered new tserver with Master: 70231b6fa29f41cc9feb6fd538393219 (127.0.97.1:39043)
I20260929 15:49:41.499145 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48637
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:49:41.535216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34623
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:41.883298 18891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:41.883666 18891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:41.883828 18891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:41.891693 18891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:41.891872 18891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:41.892064 18891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:41.902546 18891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:41.903967 18891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:41.905325 18891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:41.919819 18899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:41.919883 18896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:41.919850 18897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:41.920825 18891 server_base.cc:1034] running on GCE node
I20260929 15:49:41.921497 18891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:41.922560 18891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:41.923818 18891 hybrid_clock.cc:648] HybridClock initialized: now 1790696981923763 us; error 44 us; skew 500 ppm
I20260929 15:49:41.924226 18891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:41 dist-test-slave-7491 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:41.927769 18891 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:49:41.929739 18891 webserver.cc:460] Webserver started at http://127.0.97.2:46071/ using document root <none> and password file <none>
I20260929 15:49:41.930581 18891 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:41.930737 18891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:41.931077 18891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:41.934690 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "aea8d9d6d0fd42f99c74dcca136b624a"
format_stamp: "Formatted at 2026-09-29 15:49:41 on dist-test-slave-7491"
server_key: "e323d6ef0ee17a3194c11f2fced1ef56"
server_key_iv: "755f875fd5ce203728cbeb90ba9cce0e"
server_key_version: "encryptionkey@0"
I20260929 15:49:41.935595 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aea8d9d6d0fd42f99c74dcca136b624a"
format_stamp: "Formatted at 2026-09-29 15:49:41 on dist-test-slave-7491"
server_key: "e323d6ef0ee17a3194c11f2fced1ef56"
server_key_iv: "755f875fd5ce203728cbeb90ba9cce0e"
server_key_version: "encryptionkey@0"
I20260929 15:49:41.941663 18891 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:49:41.945791 18906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:41.947569 18891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:49:41.947808 18891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "aea8d9d6d0fd42f99c74dcca136b624a"
format_stamp: "Formatted at 2026-09-29 15:49:41 on dist-test-slave-7491"
server_key: "e323d6ef0ee17a3194c11f2fced1ef56"
server_key_iv: "755f875fd5ce203728cbeb90ba9cce0e"
server_key_version: "encryptionkey@0"
I20260929 15:49:41.948071 18891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:41.962770 18891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:41.967658 18891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:41.968022 18891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:41.969173 18891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:41.970638 18891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:41.970746 18891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:41.970996 18891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:41.971058 18891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:42.008860 18891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42991
I20260929 15:49:42.008966 19018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42991 every 8 connection(s)
I20260929 15:49:42.010381 18891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:42.012003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 18891
I20260929 15:49:42.012223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:49:42.012785 388 external_mini_cluster.cc:1428] Setting key c909fcc524cb501bbeeb3505e4fbc57c
Sep 29 15:49:42 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:42.037873 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.013750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38947 (local address 127.0.97.62:34623)
0929 15:49:42.014228 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.014232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.015495 (+ 1263us) server_negotiation.cc:407] Connection header received
0929 15:49:42.018732 (+ 3237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.018738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.018824 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.018965 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:42.021365 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.022240 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.024977 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.025426 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.025557 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.029844 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:42.029856 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:42.029863 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:42.029915 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:42.033720 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.034770 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.034779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.034786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.034972 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.035465 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.035473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.035479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.035889 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:42.036075 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.037039 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.037335 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260929 15:49:42.038650 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.014364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34623 (local address 127.0.97.2:38947)
0929 15:49:42.015300 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.015353 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.018274 (+ 2921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.019296 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.019311 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.020225 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:42.021113 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.021143 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.022415 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.022424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.024748 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.024777 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.025711 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.025720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.026087 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.027001 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:42.027026 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:42.029499 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:42.033929 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:42.033938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:42.033955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:42.034517 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:42.035117 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:42.035121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:42.035124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:42.035329 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:42.036128 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:42.036136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:42.036561 (+ 425us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.036884 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.037402 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":389,"threads_started":1}
I20260929 15:49:42.040709 19019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34623
I20260929 15:49:42.041093 19019 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:42.042032 19019 heartbeater.cc:511] Master 127.0.97.62:34623 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 15:49:42.044507 18696 ts_manager.cc:194] Registered new tserver with Master: aea8d9d6d0fd42f99c74dcca136b624a (127.0.97.2:42991)
I20260929 15:49:42.045516 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:49:42.075510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34623
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:42.426617 19025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:42.426977 19025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:42.427155 19025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:42.435001 19025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:42.435196 19025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:42.435380 19025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:42.446273 19025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:42.447933 19025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:42.449453 19025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:42.464902 19030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:42.465153 19031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:42.465305 19033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:42.465749 19025 server_base.cc:1034] running on GCE node
I20260929 15:49:42.466485 19025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:42.467923 19025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:42.469202 19025 hybrid_clock.cc:648] HybridClock initialized: now 1790696982469096 us; error 134 us; skew 500 ppm
I20260929 15:49:42.469599 19025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:42 dist-test-slave-7491 krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:42.474076 19025 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:49:42.475862 19025 webserver.cc:460] Webserver started at http://127.0.97.3:34861/ using document root <none> and password file <none>
I20260929 15:49:42.476684 19025 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:42.476843 19025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:42.477183 19025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:42.480398 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "4e4ca27ca5584175a00c2127ec5906b5"
format_stamp: "Formatted at 2026-09-29 15:49:42 on dist-test-slave-7491"
server_key: "71593b59269db84a3c28d6dfff5589f2"
server_key_iv: "168ac8bbffd49c1ed3499092391d2c80"
server_key_version: "encryptionkey@0"
I20260929 15:49:42.481249 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4e4ca27ca5584175a00c2127ec5906b5"
format_stamp: "Formatted at 2026-09-29 15:49:42 on dist-test-slave-7491"
server_key: "71593b59269db84a3c28d6dfff5589f2"
server_key_iv: "168ac8bbffd49c1ed3499092391d2c80"
server_key_version: "encryptionkey@0"
I20260929 15:49:42.486819 19025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:49:42.490799 19040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:42.492345 19025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:42.492550 19025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "4e4ca27ca5584175a00c2127ec5906b5"
format_stamp: "Formatted at 2026-09-29 15:49:42 on dist-test-slave-7491"
server_key: "71593b59269db84a3c28d6dfff5589f2"
server_key_iv: "168ac8bbffd49c1ed3499092391d2c80"
server_key_version: "encryptionkey@0"
I20260929 15:49:42.492780 19025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:42.503242 18885 heartbeater.cc:503] Master 127.0.97.62:34623 was elected leader, sending a full tablet report...
I20260929 15:49:42.518025 19025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:42.522822 19025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:42.523164 19025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:42.524295 19025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:42.525890 19025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:42.526017 19025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:42.526188 19025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:42.526268 19025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:42.563746 19025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35751
I20260929 15:49:42.563771 19152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35751 every 8 connection(s)
I20260929 15:49:42.565208 19025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:42.574239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19025
I20260929 15:49:42.574424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:49:42.575009 388 external_mini_cluster.cc:1428] Setting key 5b7311730cb792601602fcf5d57fa3d8
Sep 29 15:49:42 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:42.588433 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.568188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45077 (local address 127.0.97.62:34623)
0929 15:49:42.568673 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.568677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.569631 (+ 954us) server_negotiation.cc:407] Connection header received
0929 15:49:42.571613 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.571619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.571728 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.571884 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:42.573687 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.574994 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.577365 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.577975 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.578155 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.582096 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:42.582103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:42.582106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:42.582153 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:42.585030 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.585923 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.585927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.585930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.586064 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.586537 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.586540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.586542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.586813 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:42.586935 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.587849 (+ 914us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.588047 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260929 15:49:42.589458 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.568747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34623 (local address 127.0.97.3:45077)
0929 15:49:42.569458 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.569496 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.571252 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.572200 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.572215 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.572862 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:42.573474 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.573493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.575222 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.575227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.577203 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.577220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.578416 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.578425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.578687 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.579388 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:42.579413 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:42.581806 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:42.585226 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:42.585235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:42.585250 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:42.585782 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:42.586199 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:42.586203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:42.586205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:42.586415 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:42.586954 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:42.586961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:42.587340 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.587609 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.588053 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":270,"threads_started":1}
I20260929 15:49:42.590998 19153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34623
I20260929 15:49:42.591292 19153 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:42.591993 19153 heartbeater.cc:511] Master 127.0.97.62:34623 requested a full tablet report, sending...
I20260929 15:49:42.593612 18696 ts_manager.cc:194] Registered new tserver with Master: 4e4ca27ca5584175a00c2127ec5906b5 (127.0.97.3:35751)
I20260929 15:49:42.594486 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45077
I20260929 15:49:42.604059 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:49:42.623150 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.608521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.0.97.62:34623)
0929 15:49:42.608906 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.608914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.609317 (+ 403us) server_negotiation.cc:407] Connection header received
0929 15:49:42.609536 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.609541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.609703 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.609846 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:42.611511 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.612338 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.613682 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.614092 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.614236 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.617211 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:42.617217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:42.617220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:42.617256 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:42.619966 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.620946 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.620951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.620953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.621083 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.621576 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.621586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.621588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.621922 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:42.622216 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.622618 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.622849 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 15:49:42.628894 18696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:49:42.636330 18696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:49:42.662312 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.652312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.0.97.1:39043)
0929 15:49:42.652975 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.652985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.653015 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:49:42.653121 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.653134 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.653433 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.654207 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:42.654430 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.656545 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.659578 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.661177 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.661322 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.661575 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.661899 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.662040 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":591,"threads_started":1}
I20260929 15:49:42.662530 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.651617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39043 (local address 127.0.0.1:47446)
0929 15:49:42.652520 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.652566 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.652738 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.653678 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.653689 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.653724 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:42.654121 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.654136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.656714 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.656723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.659328 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.659345 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.661387 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.661394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.661642 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.661697 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.661855 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.662062 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":3,"thread_start_us":453,"threads_started":1}
I20260929 15:49:42.663411 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.653529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35751 (local address 127.0.0.1:59094)
0929 15:49:42.655377 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.655394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.655506 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.656419 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.656423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.656441 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:42.656794 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.656806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.658761 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.658766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.660744 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.660757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.662543 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.662550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.662731 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.662775 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.662878 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.663023 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"thread_start_us":1511,"threads_started":1}
I20260929 15:49:42.663669 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.652983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42991 (local address 127.0.0.1:59672)
0929 15:49:42.653503 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.653523 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.653666 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.654723 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.654727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.654753 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:42.655292 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.655308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.657267 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.657272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.659462 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.659474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.662869 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.662877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.663062 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.663092 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.663200 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.663345 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":244,"threads_started":1}
I20260929 15:49:42.664403 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.654069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.0.97.3:35751)
0929 15:49:42.654806 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.654818 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.655627 (+ 809us) server_negotiation.cc:407] Connection header received
0929 15:49:42.655724 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.655731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.655949 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.656126 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:42.657013 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.658502 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.660919 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.662279 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.662421 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.662723 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.662942 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.663115 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":389,"threads_started":1}
I20260929 15:49:42.664407 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.653306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.97.2:42991)
0929 15:49:42.654164 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.654171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.654195 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:49:42.654285 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.654293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.654503 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.655492 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:42.655697 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.657127 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.659645 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.661183 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.661322 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.661546 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.663892 (+ 2346us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.664082 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":607,"threads_started":1}
I20260929 15:49:42.667464 19088 tablet_service.cc:1467] Processing CreateTablet for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 (DEFAULT_TABLE table=test-table [id=d3a26c5d77d843eb87c9706deb420e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:42.667490 18820 tablet_service.cc:1467] Processing CreateTablet for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 (DEFAULT_TABLE table=test-table [id=d3a26c5d77d843eb87c9706deb420e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:42.667778 18954 tablet_service.cc:1467] Processing CreateTablet for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 (DEFAULT_TABLE table=test-table [id=d3a26c5d77d843eb87c9706deb420e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:42.668953 19088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:42.669111 18820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:42.669116 18954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:42.681599 19172 tablet_bootstrap.cc:492] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Bootstrap starting.
I20260929 15:49:42.685542 19172 tablet_bootstrap.cc:654] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:42.686662 19172 log.cc:824] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:42.688205 19173 tablet_bootstrap.cc:492] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Bootstrap starting.
I20260929 15:49:42.689051 19174 tablet_bootstrap.cc:492] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Bootstrap starting.
I20260929 15:49:42.690212 19172 tablet_bootstrap.cc:492] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: No bootstrap required, opened a new log
I20260929 15:49:42.690531 19172 ts_tablet_manager.cc:1397] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:49:42.692989 19174 tablet_bootstrap.cc:654] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:42.694484 19174 log.cc:824] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:42.696000 19173 tablet_bootstrap.cc:654] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:42.697299 19173 log.cc:824] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:42.698030 19174 tablet_bootstrap.cc:492] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: No bootstrap required, opened a new log
I20260929 15:49:42.698400 19174 ts_tablet_manager.cc:1397] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 15:49:42.699994 19173 tablet_bootstrap.cc:492] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: No bootstrap required, opened a new log
I20260929 15:49:42.700315 19173 ts_tablet_manager.cc:1397] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260929 15:49:42.701490 19172 raft_consensus.cc:348] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.701988 19172 raft_consensus.cc:374] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:42.702114 19172 raft_consensus.cc:729] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4ca27ca5584175a00c2127ec5906b5, State: Initialized, Role: FOLLOWER
I20260929 15:49:42.702834 19172 consensus_queue.cc:260] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.704998 19172 ts_tablet_manager.cc:1428] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Time spent starting tablet: real 0.014s user 0.008s sys 0.009s
I20260929 15:49:42.705638 19153 heartbeater.cc:503] Master 127.0.97.62:34623 was elected leader, sending a full tablet report...
I20260929 15:49:42.708616 19174 raft_consensus.cc:348] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.709055 19174 raft_consensus.cc:374] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:42.709183 19174 raft_consensus.cc:729] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea8d9d6d0fd42f99c74dcca136b624a, State: Initialized, Role: FOLLOWER
I20260929 15:49:42.709844 19174 consensus_queue.cc:260] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.710994 19019 heartbeater.cc:503] Master 127.0.97.62:34623 was elected leader, sending a full tablet report...
I20260929 15:49:42.711627 19174 ts_tablet_manager.cc:1428] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260929 15:49:42.710502 19173 raft_consensus.cc:348] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.712200 19173 raft_consensus.cc:374] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:42.712471 19173 raft_consensus.cc:729] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70231b6fa29f41cc9feb6fd538393219, State: Initialized, Role: FOLLOWER
I20260929 15:49:42.713169 19173 consensus_queue.cc:260] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.714148 19178 raft_consensus.cc:482] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:42.714419 19178 raft_consensus.cc:504] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.715510 19173 ts_tablet_manager.cc:1428] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260929 15:49:42.716601 19178 leader_election.cc:288] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70231b6fa29f41cc9feb6fd538393219 (127.0.97.1:39043), aea8d9d6d0fd42f99c74dcca136b624a (127.0.97.2:42991)
W20260929 15:49:42.721635 18886 tablet.cc:2369] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:42.724949 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.716923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39043 (local address 127.0.97.3:47353)
0929 15:49:42.717130 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.717151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.717277 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.718413 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.718418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.718453 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:42.718646 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.718659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.720009 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.720015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.723117 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.723136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.724364 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.724369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.724504 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.724530 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.724609 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.724722 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":29}
I20260929 15:49:42.725057 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.717884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47353 (local address 127.0.97.1:39043)
0929 15:49:42.718063 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.718068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.718087 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:49:42.718171 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.718176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.718254 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.718358 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:42.718835 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.719868 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.723305 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.724200 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.724351 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.724461 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.724660 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.724806 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:49:42.726042 18840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fe4a91b6aa4bac8e20601dc51f8fe2" candidate_uuid: "4e4ca27ca5584175a00c2127ec5906b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70231b6fa29f41cc9feb6fd538393219" is_pre_election: true
I20260929 15:49:42.726547 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.717393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42991 (local address 127.0.97.3:41631)
0929 15:49:42.718970 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:42.718989 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:42.719119 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:42.719454 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:42.719458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:42.719483 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:42.719866 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.719878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.721344 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.721349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.724030 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:42.724045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.725866 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.725871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:42.725993 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.726017 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:49:42.726093 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 15:49:42.726195 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":2,"thread_start_us":813,"threads_started":1}
I20260929 15:49:42.726588 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.717598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41631 (local address 127.0.97.2:42991)
0929 15:49:42.717818 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.717826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.719094 (+ 1268us) server_negotiation.cc:407] Connection header received
0929 15:49:42.719229 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.719234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.719315 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.719437 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:42.720067 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.721147 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.724238 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.725483 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.725657 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.725732 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.726167 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.726329 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:49:42.726722 18840 raft_consensus.cc:2454] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4ca27ca5584175a00c2127ec5906b5 in term 0.
I20260929 15:49:42.727547 19041 leader_election.cc:302] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4ca27ca5584175a00c2127ec5906b5, 70231b6fa29f41cc9feb6fd538393219; no voters:
I20260929 15:49:42.727798 18974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fe4a91b6aa4bac8e20601dc51f8fe2" candidate_uuid: "4e4ca27ca5584175a00c2127ec5906b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" is_pre_election: true
I20260929 15:49:42.728047 19178 raft_consensus.cc:2781] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:49:42.728286 19178 raft_consensus.cc:482] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:49:42.728411 19178 raft_consensus.cc:3037] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:42.728410 18974 raft_consensus.cc:2454] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4ca27ca5584175a00c2127ec5906b5 in term 0.
I20260929 15:49:42.730880 19178 raft_consensus.cc:504] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.731572 19178 leader_election.cc:288] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [CANDIDATE]: Term 1 election: Requested vote from peers 70231b6fa29f41cc9feb6fd538393219 (127.0.97.1:39043), aea8d9d6d0fd42f99c74dcca136b624a (127.0.97.2:42991)
I20260929 15:49:42.732429 18840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fe4a91b6aa4bac8e20601dc51f8fe2" candidate_uuid: "4e4ca27ca5584175a00c2127ec5906b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70231b6fa29f41cc9feb6fd538393219"
I20260929 15:49:42.732429 18974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fe4a91b6aa4bac8e20601dc51f8fe2" candidate_uuid: "4e4ca27ca5584175a00c2127ec5906b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea8d9d6d0fd42f99c74dcca136b624a"
I20260929 15:49:42.732679 18974 raft_consensus.cc:3037] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:42.732674 18840 raft_consensus.cc:3037] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:42.735610 18974 raft_consensus.cc:2454] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4ca27ca5584175a00c2127ec5906b5 in term 1.
I20260929 15:49:42.735608 18840 raft_consensus.cc:2454] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4ca27ca5584175a00c2127ec5906b5 in term 1.
I20260929 15:49:42.736137 19041 leader_election.cc:302] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4ca27ca5584175a00c2127ec5906b5, aea8d9d6d0fd42f99c74dcca136b624a; no voters:
I20260929 15:49:42.736541 19178 raft_consensus.cc:2781] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:42.737031 19178 raft_consensus.cc:686] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4ca27ca5584175a00c2127ec5906b5, State: Running, Role: LEADER
I20260929 15:49:42.737697 19178 consensus_queue.cc:237] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } }
I20260929 15:49:42.742398 18696 catalog_manager.cc:5495] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4ca27ca5584175a00c2127ec5906b5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4e4ca27ca5584175a00c2127ec5906b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4ca27ca5584175a00c2127ec5906b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35751 } health_report { overall_health: HEALTHY } } }
W20260929 15:49:42.762956 19020 tablet.cc:2369] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:42.767658 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.761883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.97.3:35751)
0929 15:49:42.762116 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.762121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.762148 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:49:42.762259 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.762263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.762367 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.762530 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:49:42.762949 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.763733 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.765278 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.765689 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.765829 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.766470 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:49:42.766995 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:49:42.767181 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.767356 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.767413 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:49:42.776891 18974 raft_consensus.cc:1261] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 1 FOLLOWER]: Refusing update from remote peer 4e4ca27ca5584175a00c2127ec5906b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:42.777043 18840 raft_consensus.cc:1261] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 1 FOLLOWER]: Refusing update from remote peer 4e4ca27ca5584175a00c2127ec5906b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:42.778007 19178 consensus_queue.cc:1035] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea8d9d6d0fd42f99c74dcca136b624a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:42.778398 19182 consensus_queue.cc:1035] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70231b6fa29f41cc9feb6fd538393219" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:42.791774 19187 mvcc.cc:205] Tried to move back new op lower bound from 7334694841442193408 to 7334694841297887232. Current Snapshot: MvccSnapshot[applied={T|T < 7334694841442193408}]
I20260929 15:49:42.791954 19185 mvcc.cc:205] Tried to move back new op lower bound from 7334694841442193408 to 7334694841297887232. Current Snapshot: MvccSnapshot[applied={T|T < 7334694841442193408}]
I20260929 15:49:42.797852 19186 mvcc.cc:205] Tried to move back new op lower bound from 7334694841442193408 to 7334694841297887232. Current Snapshot: MvccSnapshot[applied={T|T < 7334694841442193408}]
I20260929 15:49:42.811498 18695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:42.811833 18695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:42.816442 18695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3737a8797eba4bce9db4dc3752f86271: Sending DeleteTablet for 3 replicas of tablet 41fe4a91b6aa4bac8e20601dc51f8fe2
I20260929 15:49:42.818001 19088 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59094
I20260929 15:49:42.818281 18820 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47446
I20260929 15:49:42.819087 19194 tablet_replica.cc:330] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: stopping tablet replica
I20260929 15:49:42.819235 18954 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59672
I20260929 15:49:42.819626 19194 raft_consensus.cc:2229] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:49:42.820132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18757
I20260929 15:49:42.820493 19194 raft_consensus.cc:2258] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:42.822875 19194 ts_tablet_manager.cc:1905] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:42.830932 19194 ts_tablet_manager.cc:1918] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:42.831266 19194 log.cc:1196] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/41fe4a91b6aa4bac8e20601dc51f8fe2
I20260929 15:49:42.831955 19194 ts_tablet_manager.cc:1939] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 4e4ca27ca5584175a00c2127ec5906b5: Deleting consensus metadata
I20260929 15:49:42.834214 18680 catalog_manager.cc:4841] TS 4e4ca27ca5584175a00c2127ec5906b5 (127.0.97.3:35751): tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 (table test-table [id=d3a26c5d77d843eb87c9706deb420e60]) successfully deleted
I20260929 15:49:42.836817 19195 tablet_replica.cc:330] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: stopping tablet replica
I20260929 15:49:42.837374 19195 raft_consensus.cc:2229] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:42.837859 19196 tablet_replica.cc:330] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: stopping tablet replica
I20260929 15:49:42.838124 19195 raft_consensus.cc:2258] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:42.838321 19196 raft_consensus.cc:2229] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:42.838842 19196 raft_consensus.cc:2258] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:42.840844 19195 ts_tablet_manager.cc:1905] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:42.840924 19196 ts_tablet_manager.cc:1905] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:42.847576 19195 ts_tablet_manager.cc:1918] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:42.847834 19195 log.cc:1196] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/41fe4a91b6aa4bac8e20601dc51f8fe2
I20260929 15:49:42.848440 19195 ts_tablet_manager.cc:1939] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P 70231b6fa29f41cc9feb6fd538393219: Deleting consensus metadata
I20260929 15:49:42.848562 19196 ts_tablet_manager.cc:1918] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:42.848747 19196 log.cc:1196] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/41fe4a91b6aa4bac8e20601dc51f8fe2
I20260929 15:49:42.849303 19196 ts_tablet_manager.cc:1939] T 41fe4a91b6aa4bac8e20601dc51f8fe2 P aea8d9d6d0fd42f99c74dcca136b624a: Deleting consensus metadata
I20260929 15:49:42.850299 18680 catalog_manager.cc:4841] TS 70231b6fa29f41cc9feb6fd538393219 (127.0.97.1:39043): tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 (table test-table [id=d3a26c5d77d843eb87c9706deb420e60]) successfully deleted
I20260929 15:49:42.851046 18680 catalog_manager.cc:4841] TS aea8d9d6d0fd42f99c74dcca136b624a (127.0.97.2:42991): tablet 41fe4a91b6aa4bac8e20601dc51f8fe2 (table test-table [id=d3a26c5d77d843eb87c9706deb420e60]) successfully deleted
Sep 29 15:49:42 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:49:42.862287 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.821329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.0.97.1:39043)
0929 15:49:42.821514 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.821520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.837089 (+ 15569us) server_negotiation.cc:407] Connection header received
0929 15:49:42.837765 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.837771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.837887 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.837966 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:42.839854 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.840997 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.843000 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.843694 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.843814 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.855879 (+ 12065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:42.855896 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:42.855912 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:42.855967 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:42.859278 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.860057 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.860066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.860068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.860273 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.860658 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.860664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.860668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.861051 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:42.861355 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.861683 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.861914 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:49:42.863225 18880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:42.955598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18757
I20260929 15:49:42.980928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18891
Sep 29 15:49:42 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:49:42.998931 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:42.981720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.0.97.2:42991)
0929 15:49:42.981908 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:42.981912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:42.981930 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:49:42.982113 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:42.982117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:42.982189 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:42.982270 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:42.984031 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.984842 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.986718 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:42.987091 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:42.987194 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:42.992562 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:42.992584 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:42.992605 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:42.992664 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:42.995614 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.996715 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.996723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.996725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.996859 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:42.997254 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:42.997260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:42.997263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:42.997722 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:42.998048 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:42.998343 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 15:49:42.998571 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:49:42.999701 19014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:43.091272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18891
I20260929 15:49:43.109915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19025
Sep 29 15:49:43 dist-test-slave-7491 krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:49:43.131108 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:43.110916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.0.97.3:35751)
0929 15:49:43.111103 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:43.111107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:43.111138 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:49:43.111268 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:43.111272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:43.111338 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:43.111448 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:43.113431 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:43.114663 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:43.116860 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:43.117337 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:43.117450 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:43.123471 (+ 6021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:43.123487 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:43.123503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:43.123646 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:43.127338 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:43.128586 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:43.128595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:43.128597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:43.128785 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:43.129306 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:43.129310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:43.129312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:43.129839 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:43.130112 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:43.130537 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 15:49:43.130738 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:49:43.131904 19148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:43.257558 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19025
I20260929 15:49:43.274345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 18664
I20260929 15:49:43.275617 18726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:43.389421 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 18664
2026-09-29T15:49:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4542 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T15:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:43Z Disabled control of system clock
I20260929 15:49:43.469233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43157
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:49:43.819932 19205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:43.820302 19205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:43.820520 19205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:43.828397 19205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:43.828523 19205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:43.828624 19205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:43.828723 19205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:43.828804 19205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:43.838925 19205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43157
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:43.840425 19205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:43.842397 19205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:43.856300 19210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:43.856374 19213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:43.856263 19211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:43.857286 19205 server_base.cc:1034] running on GCE node
I20260929 15:49:43.858211 19205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:43.859877 19205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:43.861171 19205 hybrid_clock.cc:648] HybridClock initialized: now 1790696983861083 us; error 88 us; skew 500 ppm
I20260929 15:49:43.861768 19205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:43.865262 19205 webserver.cc:460] Webserver started at http://127.0.97.62:36699/ using document root <none> and password file <none>
I20260929 15:49:43.866096 19205 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:43.866255 19205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:43.866595 19205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:43.870030 19205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "db46190311fd423cab4c599d019aa009"
format_stamp: "Formatted at 2026-09-29 15:49:43 on dist-test-slave-7491"
server_key: "9a4a142df060d232ef54ee94ebb621ba"
server_key_iv: "32d17b3eb82d80f8a71465621df64fdb"
server_key_version: "encryptionkey@0"
I20260929 15:49:43.871013 19205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "db46190311fd423cab4c599d019aa009"
format_stamp: "Formatted at 2026-09-29 15:49:43 on dist-test-slave-7491"
server_key: "9a4a142df060d232ef54ee94ebb621ba"
server_key_iv: "32d17b3eb82d80f8a71465621df64fdb"
server_key_version: "encryptionkey@0"
I20260929 15:49:43.877332 19205 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 15:49:43.881347 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:43.882814 19205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:49:43.883042 19205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "db46190311fd423cab4c599d019aa009"
format_stamp: "Formatted at 2026-09-29 15:49:43 on dist-test-slave-7491"
server_key: "9a4a142df060d232ef54ee94ebb621ba"
server_key_iv: "32d17b3eb82d80f8a71465621df64fdb"
server_key_version: "encryptionkey@0"
I20260929 15:49:43.883285 19205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:43.922166 19205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:43.923135 19205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:43.923458 19205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:43.944099 19205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43157
I20260929 15:49:43.944132 19270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43157 every 8 connection(s)
I20260929 15:49:43.945823 19205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:43.950269 19271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:43.955442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19205
I20260929 15:49:43.955643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:43.956242 388 external_mini_cluster.cc:1428] Setting key b0603e07da4af818c57ec4bec19c0b90
I20260929 15:49:43.962863 19271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009: Bootstrap starting.
I20260929 15:49:43.966305 19271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:43.967500 19271 log.cc:824] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:43.972152 19271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009: No bootstrap required, opened a new log
I20260929 15:49:43.972347 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:43.958358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.0.97.62:43157)
0929 15:49:43.959488 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:43.959500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:43.959563 (+ 63us) server_negotiation.cc:407] Connection header received
0929 15:49:43.961567 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:43.961594 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:43.962126 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:43.962553 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:43.963171 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:43.964918 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:43.966396 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:43.967228 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:43.967460 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:43.968939 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:43.968948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:43.968964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:43.969721 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:43.969834 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:43.969865 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:43.970000 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:43.970275 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 15:49:43.970838 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":372,"threads_started":1}
I20260929 15:49:43.981678 19271 raft_consensus.cc:348] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46190311fd423cab4c599d019aa009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260929 15:49:43.982121 19271 raft_consensus.cc:374] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:43.982193 19271 raft_consensus.cc:729] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db46190311fd423cab4c599d019aa009, State: Initialized, Role: FOLLOWER
I20260929 15:49:43.982781 19271 consensus_queue.cc:260] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46190311fd423cab4c599d019aa009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260929 15:49:43.983021 19271 raft_consensus.cc:388] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:43.983132 19271 raft_consensus.cc:482] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:43.983263 19271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:43.986187 19271 raft_consensus.cc:504] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46190311fd423cab4c599d019aa009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260929 15:49:43.986725 19271 leader_election.cc:302] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db46190311fd423cab4c599d019aa009; no voters:
I20260929 15:49:43.987365 19271 leader_election.cc:288] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:43.987553 19276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:43.988749 19276 raft_consensus.cc:686] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [term 1 LEADER]: Becoming Leader. State: Replica: db46190311fd423cab4c599d019aa009, State: Running, Role: LEADER
I20260929 15:49:43.989313 19276 consensus_queue.cc:237] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46190311fd423cab4c599d019aa009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } }
I20260929 15:49:43.989823 19271 sys_catalog.cc:564] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:43.992429 19278 sys_catalog.cc:455] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db46190311fd423cab4c599d019aa009. Latest consensus state: current_term: 1 leader_uuid: "db46190311fd423cab4c599d019aa009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46190311fd423cab4c599d019aa009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } } }
I20260929 15:49:43.993201 19278 sys_catalog.cc:458] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:43.992777 19277 sys_catalog.cc:455] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db46190311fd423cab4c599d019aa009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db46190311fd423cab4c599d019aa009" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43157 } } }
I20260929 15:49:43.993768 19277 sys_catalog.cc:458] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:43.995124 19282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:44.001799 19282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:44.012679 19282 catalog_manager.cc:1347] Generated new cluster ID: 78a6045c1ba0448a94487d294d4788d2
I20260929 15:49:44.012835 19282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:44.035266 19282 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:44.036743 19282 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:44.042969 19282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009: Generated new TSK 0
I20260929 15:49:44.043993 19282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:49:44.057041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--builtin_ntp_servers=127.0.97.20:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:49:44.419008 19295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:44.419332 19295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:44.419421 19295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:44.426899 19295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:44.427066 19295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:44.427245 19295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:44.436851 19295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:44.438264 19295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:44.440117 19295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:44.454242 19301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:44.454319 19300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:44.454319 19303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:44.455681 19295 server_base.cc:1034] running on GCE node
I20260929 15:49:44.456435 19295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:44.457486 19295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:44.458678 19295 hybrid_clock.cc:648] HybridClock initialized: now 1790696984458630 us; error 38 us; skew 500 ppm
I20260929 15:49:44.459012 19295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:44.461503 19295 webserver.cc:460] Webserver started at http://127.0.97.1:40993/ using document root <none> and password file <none>
I20260929 15:49:44.462239 19295 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:44.462356 19295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:44.462697 19295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:44.465708 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "49be310d9a9d415bafb0c2f8257cd9a4"
format_stamp: "Formatted at 2026-09-29 15:49:44 on dist-test-slave-7491"
server_key: "c6165c305ea64a83777fd07af2025323"
server_key_iv: "b830b90387fafeace3535a930ebfa505"
server_key_version: "encryptionkey@0"
I20260929 15:49:44.466627 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "49be310d9a9d415bafb0c2f8257cd9a4"
format_stamp: "Formatted at 2026-09-29 15:49:44 on dist-test-slave-7491"
server_key: "c6165c305ea64a83777fd07af2025323"
server_key_iv: "b830b90387fafeace3535a930ebfa505"
server_key_version: "encryptionkey@0"
I20260929 15:49:44.472395 19295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:49:44.475960 19309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:44.477510 19295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:44.477699 19295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "49be310d9a9d415bafb0c2f8257cd9a4"
format_stamp: "Formatted at 2026-09-29 15:49:44 on dist-test-slave-7491"
server_key: "c6165c305ea64a83777fd07af2025323"
server_key_iv: "b830b90387fafeace3535a930ebfa505"
server_key_version: "encryptionkey@0"
I20260929 15:49:44.477926 19295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:44.490120 19295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:44.490950 19295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:44.491261 19295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:44.492332 19295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:44.493916 19295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:44.494009 19295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:44.494086 19295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:44.494144 19295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:44.528694 19295 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38067
I20260929 15:49:44.528741 19421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38067 every 8 connection(s)
I20260929 15:49:44.529990 19295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:44.537894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19295
I20260929 15:49:44.538082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:49:44.538614 388 external_mini_cluster.cc:1428] Setting key ec3c761a748c60a95d55fa50d8287909
I20260929 15:49:44.541929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--builtin_ntp_servers=127.0.97.20:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:49:44.545051 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:44.532737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33689 (local address 127.0.97.62:43157)
0929 15:49:44.533239 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:44.533246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:44.534519 (+ 1273us) server_negotiation.cc:407] Connection header received
0929 15:49:44.536469 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:44.536474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:44.536553 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:44.536671 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:44.537828 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:44.538939 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:44.541312 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:44.542034 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:44.542194 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:44.543999 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:44.544008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:44.544016 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:44.544045 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:44.544102 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:44.544123 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:44.544246 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:44.544477 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 15:49:44.544597 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260929 15:49:44.545982 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:44.533364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43157 (local address 127.0.97.1:33689)
0929 15:49:44.534334 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:44.534372 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:44.536075 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:44.536881 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:44.536890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:44.536990 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:44.537614 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:44.537632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:44.539135 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:44.539139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:44.541146 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:44.541161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:44.542470 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:44.542481 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:44.542749 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:44.543602 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:44.543605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:44.543664 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:44.543762 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:44.544260 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:44.544269 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:44.544275 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:49:44.544440 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 15:49:44.544783 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":304,"threads_started":1}
I20260929 15:49:44.547739 19422 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43157
I20260929 15:49:44.548110 19422 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:44.549127 19422 heartbeater.cc:511] Master 127.0.97.62:43157 requested a full tablet report, sending...
I20260929 15:49:44.551940 19236 ts_manager.cc:194] Registered new tserver with Master: 49be310d9a9d415bafb0c2f8257cd9a4 (127.0.97.1:38067)
I20260929 15:49:44.553565 19236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33689
W20260929 15:49:44.904922 19426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:44.905236 19426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:44.905385 19426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:44.913394 19426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:44.913539 19426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:44.913823 19426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:44.924769 19426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:44.926180 19426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:44.928124 19426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:44.942909 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:44.942863 19434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:44.943292 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:44.944377 19426 server_base.cc:1034] running on GCE node
I20260929 15:49:44.949198 19426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:44.950820 19426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:44.952134 19426 hybrid_clock.cc:648] HybridClock initialized: now 1790696984952037 us; error 90 us; skew 500 ppm
I20260929 15:49:44.952580 19426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:44.956091 19426 webserver.cc:460] Webserver started at http://127.0.97.2:42243/ using document root <none> and password file <none>
I20260929 15:49:44.957110 19426 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:44.957250 19426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:44.957600 19426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:44.961870 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "067030dfac4a4b50aae3e81bb319b9b1"
format_stamp: "Formatted at 2026-09-29 15:49:44 on dist-test-slave-7491"
server_key: "cea2a64a6c56fe8e9a94204c4bf87932"
server_key_iv: "3d51f1b237d8717cb7bdb281fff1d90f"
server_key_version: "encryptionkey@0"
I20260929 15:49:44.963020 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "067030dfac4a4b50aae3e81bb319b9b1"
format_stamp: "Formatted at 2026-09-29 15:49:44 on dist-test-slave-7491"
server_key: "cea2a64a6c56fe8e9a94204c4bf87932"
server_key_iv: "3d51f1b237d8717cb7bdb281fff1d90f"
server_key_version: "encryptionkey@0"
I20260929 15:49:44.969561 19426 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 15:49:44.974375 19440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:44.976290 19426 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:49:44.976471 19426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "067030dfac4a4b50aae3e81bb319b9b1"
format_stamp: "Formatted at 2026-09-29 15:49:44 on dist-test-slave-7491"
server_key: "cea2a64a6c56fe8e9a94204c4bf87932"
server_key_iv: "3d51f1b237d8717cb7bdb281fff1d90f"
server_key_version: "encryptionkey@0"
I20260929 15:49:44.976698 19426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:45.009621 19426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:45.010504 19426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:45.010751 19426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:45.011747 19426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:45.013339 19426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:45.013440 19426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:45.013516 19426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:45.013573 19426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:45.050303 19426 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33977
I20260929 15:49:45.050384 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33977 every 8 connection(s)
I20260929 15:49:45.051803 19426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:45.053754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19426
I20260929 15:49:45.053926 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:49:45.054344 388 external_mini_cluster.cc:1428] Setting key e4888c60467cd4a4b0be0a6661d25318
I20260929 15:49:45.057842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--builtin_ntp_servers=127.0.97.20:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:49:45.068632 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.055406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34937 (local address 127.0.97.62:43157)
0929 15:49:45.055874 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.055879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.060257 (+ 4378us) server_negotiation.cc:407] Connection header received
0929 15:49:45.060329 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.060333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.060411 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.060540 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:45.061826 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.062647 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.064675 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.065515 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.065704 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.067362 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:45.067373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:45.067382 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:45.067415 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:45.067477 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:45.067500 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:45.067622 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.067951 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.068092 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":258,"threads_started":1}
I20260929 15:49:45.069640 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.056068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43157 (local address 127.0.97.2:34937)
0929 15:49:45.058006 (+ 1938us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.058067 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.059816 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.060841 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.060851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.060962 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:45.061604 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.061623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.062884 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.062901 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.064479 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.064496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.065820 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.065826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.066105 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.066942 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:45.066945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:45.066993 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:45.067099 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:45.067669 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:45.067679 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:45.067687 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.067888 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.068370 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":525,"threads_started":1}
I20260929 15:49:45.072045 19553 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43157
I20260929 15:49:45.072422 19553 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:45.073482 19553 heartbeater.cc:511] Master 127.0.97.62:43157 requested a full tablet report, sending...
I20260929 15:49:45.075263 19236 ts_manager.cc:194] Registered new tserver with Master: 067030dfac4a4b50aae3e81bb319b9b1 (127.0.97.2:33977)
I20260929 15:49:45.076227 19236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34937
W20260929 15:49:45.408295 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:45.408623 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:45.408787 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:45.416322 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:45.416435 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:45.416688 19557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:45.426774 19557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:45.428226 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:45.430241 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:45.443989 19565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:45.444111 19562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:45.444134 19563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:45.445598 19557 server_base.cc:1034] running on GCE node
I20260929 15:49:45.446300 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:45.447377 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:45.448675 19557 hybrid_clock.cc:648] HybridClock initialized: now 1790696985448618 us; error 43 us; skew 500 ppm
I20260929 15:49:45.449057 19557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:45.451905 19557 webserver.cc:460] Webserver started at http://127.0.97.3:39143/ using document root <none> and password file <none>
I20260929 15:49:45.453090 19557 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:45.453243 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:45.453752 19557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:45.457242 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "d4399cd9eab249c581f51d93f03ad883"
format_stamp: "Formatted at 2026-09-29 15:49:45 on dist-test-slave-7491"
server_key: "6ac3d349dc3cbe6eee4a9577514ccaa4"
server_key_iv: "9f92695ad1e10aa26da0e8c15e39fdb5"
server_key_version: "encryptionkey@0"
I20260929 15:49:45.458217 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d4399cd9eab249c581f51d93f03ad883"
format_stamp: "Formatted at 2026-09-29 15:49:45 on dist-test-slave-7491"
server_key: "6ac3d349dc3cbe6eee4a9577514ccaa4"
server_key_iv: "9f92695ad1e10aa26da0e8c15e39fdb5"
server_key_version: "encryptionkey@0"
I20260929 15:49:45.464020 19557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:49:45.468478 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:45.470013 19557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:49:45.470229 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "d4399cd9eab249c581f51d93f03ad883"
format_stamp: "Formatted at 2026-09-29 15:49:45 on dist-test-slave-7491"
server_key: "6ac3d349dc3cbe6eee4a9577514ccaa4"
server_key_iv: "9f92695ad1e10aa26da0e8c15e39fdb5"
server_key_version: "encryptionkey@0"
I20260929 15:49:45.470486 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:45.491446 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:45.492473 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:45.492779 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:45.493950 19557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:45.495667 19557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:45.495805 19557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:45.495946 19557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:45.496057 19557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:45.532475 19557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34045
I20260929 15:49:45.532522 19683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34045 every 8 connection(s)
I20260929 15:49:45.533862 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:45.538514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19557
I20260929 15:49:45.538695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:49:45.539131 388 external_mini_cluster.cc:1428] Setting key 40e9f963f6169444c460bf5d7b66e08e
I20260929 15:49:45.547439 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.536756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52835 (local address 127.0.97.62:43157)
0929 15:49:45.536987 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.536994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.538368 (+ 1374us) server_negotiation.cc:407] Connection header received
0929 15:49:45.540663 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.540668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.540765 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.540889 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:45.542037 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.542805 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.544546 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.544976 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.545122 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.546592 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:45.546597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:45.546601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:45.546620 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:45.546654 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:45.546666 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:45.546774 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.547011 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.547122 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:49:45.549019 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.537244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43157 (local address 127.0.97.3:52835)
0929 15:49:45.538171 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.538214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.540331 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.541156 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.541165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.541261 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:45.541858 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.541875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.542974 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.542980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.544372 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.544387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.545236 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.545243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.545479 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.546221 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:45.546224 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:45.546269 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:45.546364 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:45.546776 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:45.546790 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:45.546800 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.547002 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.547458 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":285,"threads_started":1}
I20260929 15:49:45.550896 19684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43157
I20260929 15:49:45.551272 19684 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:45.552155 19684 heartbeater.cc:511] Master 127.0.97.62:43157 requested a full tablet report, sending...
I20260929 15:49:45.553943 19236 ts_manager.cc:194] Registered new tserver with Master: d4399cd9eab249c581f51d93f03ad883 (127.0.97.3:34045)
I20260929 15:49:45.554636 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:49:45.554944 19236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52835
I20260929 15:49:45.557813 19422 heartbeater.cc:503] Master 127.0.97.62:43157 was elected leader, sending a full tablet report...
I20260929 15:49:45.565843 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.559285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.0.97.62:43157)
0929 15:49:45.559484 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.559492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.559916 (+ 424us) server_negotiation.cc:407] Connection header received
0929 15:49:45.560116 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.560122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.560239 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.560331 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:45.560859 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.561611 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.563053 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.563516 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.563673 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.565029 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:45.565042 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:45.565046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:45.565066 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:45.565100 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:45.565112 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:45.565267 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.565459 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.565518 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:49:45.570701 19236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:49:45.578478 19236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:49:45.606734 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.592700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57298)
0929 15:49:45.596041 (+ 3341us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.596091 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.596812 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.598398 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.598402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.598434 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:45.599076 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.599090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.601285 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.601291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.603727 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.603738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.605639 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.605644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.605924 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.605957 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.606113 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.606322 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2999,"mutex_wait_us":150,"thread_start_us":3572,"threads_started":1}
I20260929 15:49:45.606753 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.593441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.0.97.3:34045)
0929 15:49:45.594689 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.594702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.596733 (+ 2031us) server_negotiation.cc:407] Connection header received
0929 15:49:45.597362 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.597375 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.597847 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.598084 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:45.599104 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.600639 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.603638 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.604994 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.605111 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.605382 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.606179 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.606362 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":875,"threads_started":1}
I20260929 15:49:45.606724 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.592855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.0.97.1:38067)
0929 15:49:45.595307 (+ 2452us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.595319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.596248 (+ 929us) server_negotiation.cc:407] Connection header received
0929 15:49:45.597401 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.597412 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.597870 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.598083 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:45.599261 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.601132 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.603916 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.605377 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.605512 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.605744 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.606165 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.606343 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2224,"thread_start_us":2043,"threads_started":1}
I20260929 15:49:45.606736 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.596657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.0.97.2:33977)
0929 15:49:45.597195 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.597202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.598123 (+ 921us) server_negotiation.cc:407] Connection header received
0929 15:49:45.598449 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.598457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.598645 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.598837 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:45.599486 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.601274 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.603348 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.604905 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.605065 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.605348 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.606188 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.606359 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":336,"threads_started":1}
I20260929 15:49:45.607507 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.592740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34045 (local address 127.0.0.1:59424)
0929 15:49:45.596598 (+ 3858us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.596615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.597148 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.598087 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.598094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.598126 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:45.598784 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.598800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.600802 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.600807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.603414 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.603429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.605525 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.605530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.605856 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.605987 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.606950 (+ 963us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.607103 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3725,"mutex_wait_us":48,"thread_start_us":2594,"threads_started":1}
I20260929 15:49:45.607534 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.597233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33977 (local address 127.0.0.1:40468)
0929 15:49:45.598013 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.598031 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.598243 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.598830 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.598834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.598865 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:45.599297 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.599308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.601425 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.601431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.603110 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.603130 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.605506 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.605514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.605974 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.606001 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.607000 (+ 999us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.607107 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":297,"thread_start_us":484,"threads_started":1}
I20260929 15:49:45.611402 19357 tablet_service.cc:1467] Processing CreateTablet for tablet 445b05413ba44f9491042ae5187974ad (DEFAULT_TABLE table=test-table [id=5efb95b5c0b94b68b8fd2b10de1c2c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:45.611956 19488 tablet_service.cc:1467] Processing CreateTablet for tablet 445b05413ba44f9491042ae5187974ad (DEFAULT_TABLE table=test-table [id=5efb95b5c0b94b68b8fd2b10de1c2c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:45.611955 19619 tablet_service.cc:1467] Processing CreateTablet for tablet 445b05413ba44f9491042ae5187974ad (DEFAULT_TABLE table=test-table [id=5efb95b5c0b94b68b8fd2b10de1c2c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:45.613042 19357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445b05413ba44f9491042ae5187974ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:45.613341 19619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445b05413ba44f9491042ae5187974ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:45.614190 19488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445b05413ba44f9491042ae5187974ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:45.625042 19702 tablet_bootstrap.cc:492] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Bootstrap starting.
I20260929 15:49:45.628441 19703 tablet_bootstrap.cc:492] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Bootstrap starting.
I20260929 15:49:45.629544 19702 tablet_bootstrap.cc:654] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:45.630825 19704 tablet_bootstrap.cc:492] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Bootstrap starting.
I20260929 15:49:45.630940 19702 log.cc:824] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:45.632342 19703 tablet_bootstrap.cc:654] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:45.633553 19703 log.cc:824] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:45.634574 19704 tablet_bootstrap.cc:654] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:45.634967 19702 tablet_bootstrap.cc:492] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: No bootstrap required, opened a new log
I20260929 15:49:45.635278 19702 ts_tablet_manager.cc:1397] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 15:49:45.635895 19704 log.cc:824] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:45.641072 19703 tablet_bootstrap.cc:492] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: No bootstrap required, opened a new log
I20260929 15:49:45.641407 19703 ts_tablet_manager.cc:1397] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20260929 15:49:45.642539 19704 tablet_bootstrap.cc:492] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: No bootstrap required, opened a new log
I20260929 15:49:45.642823 19704 ts_tablet_manager.cc:1397] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 15:49:45.646350 19702 raft_consensus.cc:348] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.646915 19702 raft_consensus.cc:374] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:45.647023 19702 raft_consensus.cc:729] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49be310d9a9d415bafb0c2f8257cd9a4, State: Initialized, Role: FOLLOWER
I20260929 15:49:45.647814 19702 consensus_queue.cc:260] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.651429 19704 raft_consensus.cc:348] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.651764 19702 ts_tablet_manager.cc:1428] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260929 15:49:45.651906 19704 raft_consensus.cc:374] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:45.652007 19704 raft_consensus.cc:729] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067030dfac4a4b50aae3e81bb319b9b1, State: Initialized, Role: FOLLOWER
I20260929 15:49:45.652741 19704 consensus_queue.cc:260] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.652927 19703 raft_consensus.cc:348] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.653379 19703 raft_consensus.cc:374] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:45.653482 19703 raft_consensus.cc:729] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4399cd9eab249c581f51d93f03ad883, State: Initialized, Role: FOLLOWER
I20260929 15:49:45.654065 19703 consensus_queue.cc:260] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.654740 19553 heartbeater.cc:503] Master 127.0.97.62:43157 was elected leader, sending a full tablet report...
I20260929 15:49:45.655370 19704 ts_tablet_manager.cc:1428] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 15:49:45.656804 19684 heartbeater.cc:503] Master 127.0.97.62:43157 was elected leader, sending a full tablet report...
I20260929 15:49:45.657691 19703 ts_tablet_manager.cc:1428] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Time spent starting tablet: real 0.016s user 0.010s sys 0.008s
I20260929 15:49:45.715040 19710 raft_consensus.cc:482] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:45.715430 19710 raft_consensus.cc:504] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.717486 19710 leader_election.cc:288] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49be310d9a9d415bafb0c2f8257cd9a4 (127.0.97.1:38067), 067030dfac4a4b50aae3e81bb319b9b1 (127.0.97.2:33977)
I20260929 15:49:45.724670 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.717933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35411 (local address 127.0.97.1:38067)
0929 15:49:45.718171 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.718178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.718505 (+ 327us) server_negotiation.cc:407] Connection header received
0929 15:49:45.718743 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.718750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.718852 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.718981 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:45.719491 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.720507 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.722668 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.723693 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.723791 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.723863 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.724261 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.724406 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:49:45.724677 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.717942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.97.3:35411)
0929 15:49:45.718344 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.718374 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.718564 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.719063 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.719066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.719089 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:45.719316 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.719331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.720653 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.720659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.722436 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.722498 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.723854 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.723860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.724051 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.724099 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.724208 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.724329 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":113}
I20260929 15:49:45.724880 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.718006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48135 (local address 127.0.97.2:33977)
0929 15:49:45.718307 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.718312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.719003 (+ 691us) server_negotiation.cc:407] Connection header received
0929 15:49:45.719186 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.719192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.719283 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.719389 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:45.720023 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.720884 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.723034 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.723925 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.724050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.724137 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.724468 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.724609 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 15:49:45.725319 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.718024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33977 (local address 127.0.97.3:48135)
0929 15:49:45.718874 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.718894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:45.719042 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:45.719480 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:45.719485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:45.719516 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:45.719862 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.719873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.721034 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.721038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.722827 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:45.722843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.724085 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.724089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:45.724289 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.724321 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:49:45.724413 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 15:49:45.724537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":25,"thread_start_us":454,"threads_started":1}
I20260929 15:49:45.725847 19377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445b05413ba44f9491042ae5187974ad" candidate_uuid: "d4399cd9eab249c581f51d93f03ad883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" is_pre_election: true
I20260929 15:49:45.726425 19377 raft_consensus.cc:2454] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4399cd9eab249c581f51d93f03ad883 in term 0.
I20260929 15:49:45.726765 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445b05413ba44f9491042ae5187974ad" candidate_uuid: "d4399cd9eab249c581f51d93f03ad883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067030dfac4a4b50aae3e81bb319b9b1" is_pre_election: true
I20260929 15:49:45.727247 19508 raft_consensus.cc:2454] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4399cd9eab249c581f51d93f03ad883 in term 0.
I20260929 15:49:45.727260 19574 leader_election.cc:302] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49be310d9a9d415bafb0c2f8257cd9a4, d4399cd9eab249c581f51d93f03ad883; no voters:
I20260929 15:49:45.727705 19710 raft_consensus.cc:2781] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:49:45.727867 19710 raft_consensus.cc:482] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:49:45.727986 19710 raft_consensus.cc:3037] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:45.730309 19710 raft_consensus.cc:504] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.730906 19710 leader_election.cc:288] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [CANDIDATE]: Term 1 election: Requested vote from peers 49be310d9a9d415bafb0c2f8257cd9a4 (127.0.97.1:38067), 067030dfac4a4b50aae3e81bb319b9b1 (127.0.97.2:33977)
I20260929 15:49:45.731415 19377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445b05413ba44f9491042ae5187974ad" candidate_uuid: "d4399cd9eab249c581f51d93f03ad883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49be310d9a9d415bafb0c2f8257cd9a4"
I20260929 15:49:45.731423 19508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445b05413ba44f9491042ae5187974ad" candidate_uuid: "d4399cd9eab249c581f51d93f03ad883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067030dfac4a4b50aae3e81bb319b9b1"
I20260929 15:49:45.731652 19377 raft_consensus.cc:3037] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:45.731652 19508 raft_consensus.cc:3037] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:45.735059 19377 raft_consensus.cc:2454] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4399cd9eab249c581f51d93f03ad883 in term 1.
I20260929 15:49:45.735066 19508 raft_consensus.cc:2454] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4399cd9eab249c581f51d93f03ad883 in term 1.
I20260929 15:49:45.735574 19574 leader_election.cc:302] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49be310d9a9d415bafb0c2f8257cd9a4, d4399cd9eab249c581f51d93f03ad883; no voters:
I20260929 15:49:45.735903 19710 raft_consensus.cc:2781] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:45.736414 19710 raft_consensus.cc:686] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 1 LEADER]: Becoming Leader. State: Replica: d4399cd9eab249c581f51d93f03ad883, State: Running, Role: LEADER
I20260929 15:49:45.736936 19710 consensus_queue.cc:237] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } }
I20260929 15:49:45.741989 19235 catalog_manager.cc:5495] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4399cd9eab249c581f51d93f03ad883 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d4399cd9eab249c581f51d93f03ad883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4399cd9eab249c581f51d93f03ad883" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 } health_report { overall_health: UNKNOWN } } }
I20260929 15:49:45.766434 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.760956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.0.97.3:34045)
0929 15:49:45.761152 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.761157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.761234 (+ 77us) server_negotiation.cc:407] Connection header received
0929 15:49:45.761397 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.761400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.761498 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.761648 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:49:45.761969 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.762699 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.764260 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.764695 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.764798 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.765362 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:49:45.765879 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:49:45.766015 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.766187 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.766227 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:49:45.776988 19377 raft_consensus.cc:1261] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 1 FOLLOWER]: Refusing update from remote peer d4399cd9eab249c581f51d93f03ad883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:45.776988 19508 raft_consensus.cc:1261] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 1 FOLLOWER]: Refusing update from remote peer d4399cd9eab249c581f51d93f03ad883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:45.778198 19712 consensus_queue.cc:1035] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067030dfac4a4b50aae3e81bb319b9b1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:45.778714 19710 consensus_queue.cc:1035] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49be310d9a9d415bafb0c2f8257cd9a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:49:45.782636 19423 tablet.cc:2369] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:49:45.785665 19685 tablet.cc:2369] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:45.797930 19715 mvcc.cc:205] Tried to move back new op lower bound from 7334694853728776192 to 7334694853582528512. Current Snapshot: MvccSnapshot[applied={T|T < 7334694853728776192}]
I20260929 15:49:45.798501 19716 mvcc.cc:205] Tried to move back new op lower bound from 7334694853728776192 to 7334694853582528512. Current Snapshot: MvccSnapshot[applied={T|T < 7334694853728776192}]
I20260929 15:49:45.802482 19717 mvcc.cc:205] Tried to move back new op lower bound from 7334694853728776192 to 7334694853582528512. Current Snapshot: MvccSnapshot[applied={T|T < 7334694853728776192}]
I20260929 15:49:45.814504 19235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:45.814841 19235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:45.818840 19235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db46190311fd423cab4c599d019aa009: Sending DeleteTablet for 3 replicas of tablet 445b05413ba44f9491042ae5187974ad
I20260929 15:49:45.820274 19619 tablet_service.cc:1514] Processing DeleteTablet for tablet 445b05413ba44f9491042ae5187974ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:45 UTC) from {username='slave'} at 127.0.0.1:59424
I20260929 15:49:45.820267 19357 tablet_service.cc:1514] Processing DeleteTablet for tablet 445b05413ba44f9491042ae5187974ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:45 UTC) from {username='slave'} at 127.0.0.1:57298
I20260929 15:49:45.820559 19488 tablet_service.cc:1514] Processing DeleteTablet for tablet 445b05413ba44f9491042ae5187974ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:45 UTC) from {username='slave'} at 127.0.0.1:40468
I20260929 15:49:45.823082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19295
I20260929 15:49:45.830986 19726 tablet_replica.cc:330] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: stopping tablet replica
I20260929 15:49:45.831758 19726 raft_consensus.cc:2229] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:45.832626 19726 raft_consensus.cc:2258] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:45.833856 19725 tablet_replica.cc:330] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: stopping tablet replica
I20260929 15:49:45.834648 19724 tablet_replica.cc:330] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: stopping tablet replica
I20260929 15:49:45.835176 19724 raft_consensus.cc:2229] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:49:45.836256 19725 raft_consensus.cc:2229] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:45.836844 19725 raft_consensus.cc:2258] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:45.837177 19724 raft_consensus.cc:2258] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:45.837539 19726 ts_tablet_manager.cc:1905] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:45.839761 19725 ts_tablet_manager.cc:1905] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:45.841920 19724 ts_tablet_manager.cc:1905] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:45.845500 19726 ts_tablet_manager.cc:1918] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:45.845734 19726 log.cc:1196] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/445b05413ba44f9491042ae5187974ad
I20260929 15:49:45.846315 19726 ts_tablet_manager.cc:1939] T 445b05413ba44f9491042ae5187974ad P 067030dfac4a4b50aae3e81bb319b9b1: Deleting consensus metadata
I20260929 15:49:45.848440 19223 catalog_manager.cc:4841] TS 067030dfac4a4b50aae3e81bb319b9b1 (127.0.97.2:33977): tablet 445b05413ba44f9491042ae5187974ad (table test-table [id=5efb95b5c0b94b68b8fd2b10de1c2c40]) successfully deleted
I20260929 15:49:45.850018 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.825326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.0.97.1:38067)
0929 15:49:45.825521 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.825527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.840935 (+ 15408us) server_negotiation.cc:407] Connection header received
0929 15:49:45.841471 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.841477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.841585 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.841663 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:45.842376 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.843591 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.845634 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.846295 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.846404 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.848044 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:45.848059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:45.848085 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:45.848978 (+ 893us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:45.849045 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:45.849073 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:45.849184 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.849457 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.849587 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:49:45.851358 19417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:45.855593 19724 ts_tablet_manager.cc:1918] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:45.855875 19724 log.cc:1196] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/445b05413ba44f9491042ae5187974ad
I20260929 15:49:45.856544 19724 ts_tablet_manager.cc:1939] T 445b05413ba44f9491042ae5187974ad P d4399cd9eab249c581f51d93f03ad883: Deleting consensus metadata
I20260929 15:49:45.858327 19223 catalog_manager.cc:4841] TS d4399cd9eab249c581f51d93f03ad883 (127.0.97.3:34045): tablet 445b05413ba44f9491042ae5187974ad (table test-table [id=5efb95b5c0b94b68b8fd2b10de1c2c40]) successfully deleted
I20260929 15:49:45.953279 19725 ts_tablet_manager.cc:1918] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:45.953569 19725 log.cc:1196] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/445b05413ba44f9491042ae5187974ad
I20260929 15:49:45.954151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19295
I20260929 15:49:45.954218 19725 ts_tablet_manager.cc:1939] T 445b05413ba44f9491042ae5187974ad P 49be310d9a9d415bafb0c2f8257cd9a4: Deleting consensus metadata
W20260929 15:49:45.973735 19222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38067 (error 108)
I20260929 15:49:45.973856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19426
I20260929 15:49:45.988649 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.975998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.0.97.2:33977)
0929 15:49:45.976273 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:45.976281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:45.976306 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:49:45.976391 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:45.976400 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:45.976560 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:45.976691 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:45.977650 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.979225 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.982462 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:45.983412 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:45.983580 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:45.985610 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:45.985628 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:45.985654 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:45.986918 (+ 1264us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:45.987013 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:45.987053 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:45.987238 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:45.987918 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 15:49:45.988058 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 15:49:45.988662 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:45.988204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57318)
0929 15:49:45.988366 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:45.988485 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:49:45.989033 19222 catalog_manager.cc:4584] TS 49be310d9a9d415bafb0c2f8257cd9a4 (127.0.97.1:38067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 445b05413ba44f9491042ae5187974ad: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
I20260929 15:49:45.989724 19548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:46.042801 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.042178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57334)
0929 15:49:46.042479 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.042589 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 15:49:46.044062 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.043618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57340)
0929 15:49:46.043798 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.043888 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 15:49:46.082590 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19426
I20260929 15:49:46.101174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19557
I20260929 15:49:46.104835 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.104323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57352)
0929 15:49:46.104549 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.104654 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 15:49:46.105894 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.105566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57366)
0929 15:49:46.105711 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.105786 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:49:46.109679 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.102005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.0.97.3:34045)
0929 15:49:46.102190 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:46.102194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:46.102213 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:49:46.102428 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:46.102431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:46.102581 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:46.102791 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:46.103256 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:46.104267 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:46.106130 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:46.106597 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:46.106709 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:46.108044 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:46.108053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:46.108073 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:46.108861 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:46.108916 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:46.108940 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:46.109023 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:46.109217 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 15:49:46.109299 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:49:46.110375 19679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:46.186656 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.186068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57376)
0929 15:49:46.186383 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.186482 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260929 15:49:46.187849 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.187440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57380)
0929 15:49:46.187646 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.187741 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 15:49:46.234849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19557
I20260929 15:49:46.254906 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19205
I20260929 15:49:46.255990 19266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:46.368099 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:46.367384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38067 (local address 127.0.0.1:57396)
0929 15:49:46.367799 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:46.367918 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260929 15:49:46.368304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19205
2026-09-29T15:49:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2978 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:49:46 dist-test-slave-7491 krb5kdc[19735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:49:46 dist-test-slave-7491 krb5kdc[19735](info): set up 2 sockets
Sep 29 15:49:46 dist-test-slave-7491 krb5kdc[19735](info): commencing operation
krb5kdc: starting...
W20260929 15:49:47.480513 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:49:47 dist-test-slave-7491 krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:49:47.741487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41713
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:48.089800 19748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:48.090112 19748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:48.090281 19748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:48.097638 19748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:48.097765 19748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:48.097859 19748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:48.097971 19748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:48.098064 19748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:48.108438 19748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:48.109845 19748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:48.111141 19748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:48.123481 19753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:48.123553 19754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:48.123461 19756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:48.124619 19748 server_base.cc:1034] running on GCE node
I20260929 15:49:48.125470 19748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:48.127027 19748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:48.128319 19748 hybrid_clock.cc:648] HybridClock initialized: now 1790696988128265 us; error 34 us; skew 500 ppm
I20260929 15:49:48.128741 19748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:48 dist-test-slave-7491 krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:48.135206 19748 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:49:48.136958 19748 webserver.cc:460] Webserver started at http://127.0.97.62:37699/ using document root <none> and password file <none>
I20260929 15:49:48.137742 19748 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:48.137912 19748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:48.138244 19748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:48.141469 19748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "1d5fca0514e840009cc64f56816eb7db"
format_stamp: "Formatted at 2026-09-29 15:49:48 on dist-test-slave-7491"
server_key: "5358132a4e7d695bdea3156f44f72e63"
server_key_iv: "fa9ce37e875884778bc9c7eafb2039dd"
server_key_version: "encryptionkey@0"
I20260929 15:49:48.142266 19748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "1d5fca0514e840009cc64f56816eb7db"
format_stamp: "Formatted at 2026-09-29 15:49:48 on dist-test-slave-7491"
server_key: "5358132a4e7d695bdea3156f44f72e63"
server_key_iv: "fa9ce37e875884778bc9c7eafb2039dd"
server_key_version: "encryptionkey@0"
I20260929 15:49:48.147718 19748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 15:49:48.151463 19763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:48.152923 19748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:48.153136 19748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "1d5fca0514e840009cc64f56816eb7db"
format_stamp: "Formatted at 2026-09-29 15:49:48 on dist-test-slave-7491"
server_key: "5358132a4e7d695bdea3156f44f72e63"
server_key_iv: "fa9ce37e875884778bc9c7eafb2039dd"
server_key_version: "encryptionkey@0"
I20260929 15:49:48.153362 19748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:48.182145 19748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:48.186748 19748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:48.187090 19748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:48.206563 19748 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41713
I20260929 15:49:48.206626 19814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41713 every 8 connection(s)
I20260929 15:49:48.208081 19748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:48.210801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19748
I20260929 15:49:48.211009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:48.211440 388 external_mini_cluster.cc:1428] Setting key 7972390064574371f4893f456edd0449
I20260929 15:49:48.213099 19815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:49:48 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:48.227980 19815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db: Bootstrap starting.
I20260929 15:49:48.232576 19815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:48.233870 19815 log.cc:824] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:48.237505 19815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db: No bootstrap required, opened a new log
I20260929 15:49:48.237625 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:48.213627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.0.97.62:41713)
0929 15:49:48.214513 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:48.214528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:48.214582 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:49:48.216733 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:48.216764 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:48.217360 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:48.217849 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:48.219197 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:48.221336 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:48.222655 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:48.223708 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:48.223901 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:48.228527 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:48.228540 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:48.228563 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:48.228623 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:48.232813 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:48.233656 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:48.233668 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:48.233678 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:48.233917 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:48.234336 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:48.234342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:48.234348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:48.235033 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:48.235306 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:48.235781 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 15:49:48.236291 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":378,"threads_started":1}
I20260929 15:49:48.245929 19815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5fca0514e840009cc64f56816eb7db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41713 } }
I20260929 15:49:48.246325 19815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:48.246399 19815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5fca0514e840009cc64f56816eb7db, State: Initialized, Role: FOLLOWER
I20260929 15:49:48.246906 19815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5fca0514e840009cc64f56816eb7db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41713 } }
I20260929 15:49:48.247103 19815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:48.247201 19815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:48.247320 19815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:48.250058 19815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5fca0514e840009cc64f56816eb7db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41713 } }
I20260929 15:49:48.250559 19815 leader_election.cc:302] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d5fca0514e840009cc64f56816eb7db; no voters:
I20260929 15:49:48.251101 19815 leader_election.cc:288] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:48.251268 19820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:48.252154 19820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [term 1 LEADER]: Becoming Leader. State: Replica: 1d5fca0514e840009cc64f56816eb7db, State: Running, Role: LEADER
I20260929 15:49:48.252739 19820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5fca0514e840009cc64f56816eb7db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41713 } }
I20260929 15:49:48.253661 19815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:48.256801 19821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d5fca0514e840009cc64f56816eb7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5fca0514e840009cc64f56816eb7db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41713 } } }
I20260929 15:49:48.257165 19821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:48.256719 19822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d5fca0514e840009cc64f56816eb7db. Latest consensus state: current_term: 1 leader_uuid: "1d5fca0514e840009cc64f56816eb7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d5fca0514e840009cc64f56816eb7db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41713 } } }
I20260929 15:49:48.259383 19822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:48.260890 19827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:48.265717 19827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:48.275400 19827 catalog_manager.cc:1347] Generated new cluster ID: 435abaa7812a4bc69a188bad90d1767f
I20260929 15:49:48.275547 19827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:48.292848 19827 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:48.294852 19827 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:48.313444 19827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db: Generated new TSK 0
I20260929 15:49:48.314378 19827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 15:49:48.387349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41713
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:48.756060 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:48.756454 19841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:48.756657 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:48.764801 19841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:48.764961 19841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:48.765146 19841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:48.776687 19841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:48.778064 19841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:48.779390 19841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:48.793485 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:48.793524 19846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:48.793520 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:48.794658 19841 server_base.cc:1034] running on GCE node
I20260929 15:49:48.795415 19841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:48.796538 19841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:48.797787 19841 hybrid_clock.cc:648] HybridClock initialized: now 1790696988797739 us; error 58 us; skew 500 ppm
I20260929 15:49:48.798135 19841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:48 dist-test-slave-7491 krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:48.801643 19841 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 15:49:48.803387 19841 webserver.cc:460] Webserver started at http://127.0.97.1:45969/ using document root <none> and password file <none>
I20260929 15:49:48.804325 19841 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:48.804458 19841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:48.804739 19841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:48.807787 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "200397de790240ffb1311b46ea8d8c59"
format_stamp: "Formatted at 2026-09-29 15:49:48 on dist-test-slave-7491"
server_key: "ab2e9e53bf780ae57b4388b468e685f1"
server_key_iv: "45449db033a7b7751f1863b118874f11"
server_key_version: "encryptionkey@0"
I20260929 15:49:48.808681 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "200397de790240ffb1311b46ea8d8c59"
format_stamp: "Formatted at 2026-09-29 15:49:48 on dist-test-slave-7491"
server_key: "ab2e9e53bf780ae57b4388b468e685f1"
server_key_iv: "45449db033a7b7751f1863b118874f11"
server_key_version: "encryptionkey@0"
I20260929 15:49:48.814445 19841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:49:48.818570 19856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:48.820304 19841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:48.820477 19841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "200397de790240ffb1311b46ea8d8c59"
format_stamp: "Formatted at 2026-09-29 15:49:48 on dist-test-slave-7491"
server_key: "ab2e9e53bf780ae57b4388b468e685f1"
server_key_iv: "45449db033a7b7751f1863b118874f11"
server_key_version: "encryptionkey@0"
I20260929 15:49:48.820672 19841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:48.851188 19841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:48.856815 19841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:48.857148 19841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:48.858242 19841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:48.859671 19841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:48.859752 19841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:48.859882 19841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:48.859951 19841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:48.896112 19841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37663
I20260929 15:49:48.896322 19968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37663 every 8 connection(s)
I20260929 15:49:48.897506 19841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:48.897854 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19841
I20260929 15:49:48.898049 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:49:48.898633 388 external_mini_cluster.cc:1428] Setting key 8104b479955220cf5169a29e42ccafdb
Sep 29 15:49:48 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:48.920264 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:48.900305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36367 (local address 127.0.97.62:41713)
0929 15:49:48.900941 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:48.900948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:48.901948 (+ 1000us) server_negotiation.cc:407] Connection header received
0929 15:49:48.903846 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:48.903855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:48.903949 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:48.904087 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:48.905693 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:48.906905 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:48.908652 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:48.909131 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:48.909240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:48.913945 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:48.913951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:48.913955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:48.913995 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:48.916842 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:48.917734 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:48.917738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:48.917741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:48.917875 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:48.918362 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:48.918366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:48.918368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:48.918653 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:48.918821 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:48.919592 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 15:49:48.919848 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":406,"threads_started":1}
I20260929 15:49:48.921386 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:48.900941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41713 (local address 127.0.97.1:36367)
0929 15:49:48.901765 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:48.901817 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:48.903552 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:48.904367 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:48.904376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:48.904904 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:48.905489 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:48.905507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:48.907087 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:48.907090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:48.908494 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:48.908507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:48.909389 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:48.909397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:48.909743 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:48.910563 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:48.910599 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:48.913648 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:48.917028 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:48.917038 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:48.917067 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:48.917561 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:48.917999 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:48.918004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:48.918008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:48.918220 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:48.918793 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:48.918801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:48.919179 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 15:49:48.919456 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 15:49:48.920053 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":332,"threads_started":1}
I20260929 15:49:48.923413 19969 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41713
I20260929 15:49:48.923758 19969 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:48.924675 19969 heartbeater.cc:511] Master 127.0.97.62:41713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 15:49:48.927193 19780 ts_manager.cc:194] Registered new tserver with Master: 200397de790240ffb1311b46ea8d8c59 (127.0.97.1:37663)
I20260929 15:49:48.928845 19780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 15:49:48.957489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41713
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:49.301754 19975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:49.302207 19975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:49.302333 19975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:49.310377 19975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:49.310544 19975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:49.310673 19975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:49.321872 19975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:49.323287 19975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:49.324724 19975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:49.339759 19980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:49.339644 19981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:49.339738 19983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:49.346004 19975 server_base.cc:1034] running on GCE node
I20260929 15:49:49.346817 19975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:49.347999 19975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:49.349239 19975 hybrid_clock.cc:648] HybridClock initialized: now 1790696989349182 us; error 46 us; skew 500 ppm
I20260929 15:49:49.349643 19975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:49 dist-test-slave-7491 krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:49.353502 19975 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 15:49:49.355438 19975 webserver.cc:460] Webserver started at http://127.0.97.2:35379/ using document root <none> and password file <none>
I20260929 15:49:49.356524 19975 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:49.356696 19975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:49.357107 19975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:49.360777 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "d5ff71b312f2484593e114af3ed3f4d1"
format_stamp: "Formatted at 2026-09-29 15:49:49 on dist-test-slave-7491"
server_key: "507e75c0933527a5a04ab66fc5d62928"
server_key_iv: "507cc01a766b3cfe4dbf683ac1822370"
server_key_version: "encryptionkey@0"
I20260929 15:49:49.361881 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d5ff71b312f2484593e114af3ed3f4d1"
format_stamp: "Formatted at 2026-09-29 15:49:49 on dist-test-slave-7491"
server_key: "507e75c0933527a5a04ab66fc5d62928"
server_key_iv: "507cc01a766b3cfe4dbf683ac1822370"
server_key_version: "encryptionkey@0"
I20260929 15:49:49.367841 19975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:49:49.371618 19990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:49.373019 19975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:49.373224 19975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "d5ff71b312f2484593e114af3ed3f4d1"
format_stamp: "Formatted at 2026-09-29 15:49:49 on dist-test-slave-7491"
server_key: "507e75c0933527a5a04ab66fc5d62928"
server_key_iv: "507cc01a766b3cfe4dbf683ac1822370"
server_key_version: "encryptionkey@0"
I20260929 15:49:49.373456 19975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:49.389999 19975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:49.394631 19975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:49.395031 19975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:49.396405 19975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:49.398340 19975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:49.398466 19975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:49.398633 19975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:49.398706 19975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:49.436774 19975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35779
I20260929 15:49:49.436753 20102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35779 every 8 connection(s)
I20260929 15:49:49.438179 19975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:49.445883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 19975
I20260929 15:49:49.446105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:49:49.446599 388 external_mini_cluster.cc:1428] Setting key 7a545feab91f0d8f8a609c45effc0302
Sep 29 15:49:49 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:49.461694 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:49.440914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58813 (local address 127.0.97.62:41713)
0929 15:49:49.441430 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:49.441435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:49.442511 (+ 1076us) server_negotiation.cc:407] Connection header received
0929 15:49:49.444540 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:49.444547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:49.444630 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:49.444734 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:49.446505 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:49.447499 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:49.449586 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:49.450035 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:49.450186 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:49.453966 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:49.453971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:49.453975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:49.454080 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:49.457555 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:49.458746 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:49.458751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:49.458753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:49.458903 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:49.459430 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:49.459434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:49.459437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:49.459955 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:49.460194 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:49.460987 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 15:49:49.461275 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260929 15:49:49.462719 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:49.441461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41713 (local address 127.0.97.2:58813)
0929 15:49:49.442318 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:49.442372 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:49.444151 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:49.445013 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:49.445022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:49.445572 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:49.446298 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:49.446320 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:49.447724 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:49.447730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:49.449414 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:49.449429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:49.450307 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:49.450314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:49.450575 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:49.451325 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:49.451349 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:49.453679 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:49.457776 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:49.457784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:49.457807 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:49.458405 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:49.459048 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:49.459053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:49.459056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:49.459283 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:49.460108 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:49.460149 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:49.460572 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 15:49:49.460866 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 15:49:49.461510 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":352,"threads_started":1}
I20260929 15:49:49.464699 20103 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41713
I20260929 15:49:49.465055 20103 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:49.465863 20103 heartbeater.cc:511] Master 127.0.97.62:41713 requested a full tablet report, sending...
I20260929 15:49:49.467697 19780 ts_manager.cc:194] Registered new tserver with Master: d5ff71b312f2484593e114af3ed3f4d1 (127.0.97.2:35779)
I20260929 15:49:49.468757 19780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58813
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 15:49:49.508736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41713
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:49.878068 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:49.878414 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:49.878513 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:49.886270 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:49.886422 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:49.886540 20109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:49.898300 20109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:49.899744 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:49.901193 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:49.914965 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:49.915061 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:49.915042 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:49.916283 20109 server_base.cc:1034] running on GCE node
I20260929 15:49:49.917210 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:49.918326 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:49.919627 20109 hybrid_clock.cc:648] HybridClock initialized: now 1790696989919537 us; error 68 us; skew 500 ppm
I20260929 15:49:49.920037 20109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:49 dist-test-slave-7491 krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:49.923518 20109 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 15:49:49.925520 20109 webserver.cc:460] Webserver started at http://127.0.97.3:34163/ using document root <none> and password file <none>
I20260929 15:49:49.926290 20109 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:49.926396 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:49.926671 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:49.929817 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "b0c35aedb5364eb8a5bba11d085cb5eb"
format_stamp: "Formatted at 2026-09-29 15:49:49 on dist-test-slave-7491"
server_key: "191f4ad1b450bca71f8ce7c7953fe182"
server_key_iv: "cfe44a59a6a99e037e2b77d1029f61e5"
server_key_version: "encryptionkey@0"
I20260929 15:49:49.930596 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b0c35aedb5364eb8a5bba11d085cb5eb"
format_stamp: "Formatted at 2026-09-29 15:49:49 on dist-test-slave-7491"
server_key: "191f4ad1b450bca71f8ce7c7953fe182"
server_key_iv: "cfe44a59a6a99e037e2b77d1029f61e5"
server_key_version: "encryptionkey@0"
I20260929 15:49:49.932040 19969 heartbeater.cc:503] Master 127.0.97.62:41713 was elected leader, sending a full tablet report...
I20260929 15:49:49.936962 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 15:49:49.940785 20124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:49.942262 20109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:49:49.942413 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "b0c35aedb5364eb8a5bba11d085cb5eb"
format_stamp: "Formatted at 2026-09-29 15:49:49 on dist-test-slave-7491"
server_key: "191f4ad1b450bca71f8ce7c7953fe182"
server_key_iv: "cfe44a59a6a99e037e2b77d1029f61e5"
server_key_version: "encryptionkey@0"
I20260929 15:49:49.942605 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:49.963445 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:49.968430 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:49.968777 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:49.969954 20109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:49.971509 20109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:49.971607 20109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:49.971750 20109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:49.971807 20109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:50.009444 20109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35595
I20260929 15:49:50.009495 20236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35595 every 8 connection(s)
I20260929 15:49:50.010795 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:50.011710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20109
I20260929 15:49:50.011904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:49:50.012475 388 external_mini_cluster.cc:1428] Setting key 333560fb9e7a968d35a6cdedbf15cba8
Sep 29 15:49:50 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:49:50.035526 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.014705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39301 (local address 127.0.97.62:41713)
0929 15:49:50.016891 (+ 2186us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.016895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.016913 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:49:50.018090 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.018095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.018167 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.018282 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:50.020050 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.020948 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.022895 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.023342 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.023470 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.028292 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:50.028298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:50.028303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:50.028362 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:50.031412 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.032377 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.032382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.032385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.032537 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.033079 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.033083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.033085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.033395 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:50.033551 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.034810 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.035100 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":1958,"threads_started":1}
I20260929 15:49:50.036367 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.014758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41713 (local address 127.0.97.3:39301)
0929 15:49:50.015773 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.015826 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.017741 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.018586 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.018596 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.019169 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:50.019815 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.019836 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.021144 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.021149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.022684 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.022702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.023606 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.023612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.023864 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.024768 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:49:50.024802 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:50.027970 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:50.031609 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:50.031618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:50.031634 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:50.032146 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:50.032679 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:49:50.032683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:49:50.032685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:49:50.032898 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:49:50.033661 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:50.033671 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:50.034214 (+ 543us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.034507 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.034982 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":395,"threads_started":1}
I20260929 15:49:50.038028 20237 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41713
I20260929 15:49:50.038373 20237 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:50.039191 20237 heartbeater.cc:511] Master 127.0.97.62:41713 requested a full tablet report, sending...
I20260929 15:49:50.041172 19779 ts_manager.cc:194] Registered new tserver with Master: b0c35aedb5364eb8a5bba11d085cb5eb (127.0.97.3:35595)
I20260929 15:49:50.042333 19779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39301
I20260929 15:49:50.042408 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:49:50.061509 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.046975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.0.97.62:41713)
0929 15:49:50.047255 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.047262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.047551 (+ 289us) server_negotiation.cc:407] Connection header received
0929 15:49:50.047720 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.047724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.047790 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.047881 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:50.049495 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.050275 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.051637 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.052081 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.052288 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.055123 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:50.055131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:50.055209 (+ 78us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:50.055272 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:50.058310 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.059126 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.059130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.059133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.059266 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.059683 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.059691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.059698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.060078 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:50.060354 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.060918 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.061195 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 15:49:50.066322 19779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:49:50.073803 19779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:49:50.112263 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.088137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35595 (local address 127.0.0.1:45262)
0929 15:49:50.100319 (+ 12182us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.100395 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.100692 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.101781 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.101793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.101841 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.102451 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.102469 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.104906 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.104914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.107880 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.107897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.111270 (+ 3373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.111275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.111411 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.111446 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.111606 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.111773 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11785,"thread_start_us":16658,"threads_started":1}
I20260929 15:49:50.112746 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.091141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35779 (local address 127.0.0.1:38160)
0929 15:49:50.105136 (+ 13995us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.105155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.105329 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.106280 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.106285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.106310 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.106690 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.106697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.108033 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.108039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.110188 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.110203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.111937 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.111948 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.112123 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.112190 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.112289 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.112448 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13837,"thread_start_us":13631,"threads_started":1}
I20260929 15:49:50.113529 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.088393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.97.3:35595)
0929 15:49:50.099734 (+ 11341us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.099747 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.100968 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 15:49:50.101123 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.101135 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.101473 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.102717 (+ 1244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.103059 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.104726 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.108319 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.110814 (+ 2495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.110945 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.111161 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.112977 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.113195 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11127,"thread_start_us":10926,"threads_started":1}
I20260929 15:49:50.113719 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.091762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.0.97.2:35779)
0929 15:49:50.104546 (+ 12784us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.104560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.105279 (+ 719us) server_negotiation.cc:407] Connection header received
0929 15:49:50.105573 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.105581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.105853 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.106045 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.106860 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.107864 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.110414 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.111718 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.111864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.112048 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.113205 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.113393 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12575,"thread_start_us":12458,"threads_started":1}
I20260929 15:49:50.113940 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.088223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37663 (local address 127.0.0.1:48540)
0929 15:49:50.103195 (+ 14972us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.103216 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.103338 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.105394 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.105403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.105436 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.105999 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.106012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.108444 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.108449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.110782 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.110798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.113032 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.113040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.113215 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.113259 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.113344 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.113483 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14819,"mutex_wait_us":82,"thread_start_us":11572,"threads_started":1}
I20260929 15:49:50.114665 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.088738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.97.1:37663)
0929 15:49:50.103964 (+ 15226us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.103973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.104004 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:49:50.104105 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.104114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.104421 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.104632 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.106241 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.108075 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.110983 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.112519 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.112662 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.112884 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.114254 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.114389 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15009,"thread_start_us":14860,"threads_started":1}
I20260929 15:49:50.117942 20172 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc28256155c412886630ed048c98279 (DEFAULT_TABLE table=test-table [id=0c6e80d8a401423e98516433d28c7b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:50.118085 20038 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc28256155c412886630ed048c98279 (DEFAULT_TABLE table=test-table [id=0c6e80d8a401423e98516433d28c7b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:50.118180 19904 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc28256155c412886630ed048c98279 (DEFAULT_TABLE table=test-table [id=0c6e80d8a401423e98516433d28c7b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:50.119570 19904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc28256155c412886630ed048c98279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:50.119596 20038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc28256155c412886630ed048c98279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:50.119586 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc28256155c412886630ed048c98279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:50.133375 20256 tablet_bootstrap.cc:492] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Bootstrap starting.
I20260929 15:49:50.135512 20257 tablet_bootstrap.cc:492] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Bootstrap starting.
I20260929 15:49:50.138084 20256 tablet_bootstrap.cc:654] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:50.139302 20257 tablet_bootstrap.cc:654] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:50.139613 20256 log.cc:824] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:50.140687 20257 log.cc:824] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:50.142235 20258 tablet_bootstrap.cc:492] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Bootstrap starting.
I20260929 15:49:50.143518 20256 tablet_bootstrap.cc:492] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: No bootstrap required, opened a new log
I20260929 15:49:50.143889 20256 ts_tablet_manager.cc:1397] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260929 15:49:50.144567 20257 tablet_bootstrap.cc:492] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: No bootstrap required, opened a new log
I20260929 15:49:50.144951 20257 ts_tablet_manager.cc:1397] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260929 15:49:50.146207 20258 tablet_bootstrap.cc:654] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:50.147720 20258 log.cc:824] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:50.151597 20258 tablet_bootstrap.cc:492] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: No bootstrap required, opened a new log
I20260929 15:49:50.151913 20258 ts_tablet_manager.cc:1397] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260929 15:49:50.154739 20257 raft_consensus.cc:348] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.155174 20257 raft_consensus.cc:374] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:50.155270 20257 raft_consensus.cc:729] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c35aedb5364eb8a5bba11d085cb5eb, State: Initialized, Role: FOLLOWER
I20260929 15:49:50.154774 20256 raft_consensus.cc:348] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.155570 20256 raft_consensus.cc:374] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:50.155727 20256 raft_consensus.cc:729] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200397de790240ffb1311b46ea8d8c59, State: Initialized, Role: FOLLOWER
I20260929 15:49:50.156018 20257 consensus_queue.cc:260] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.156545 20256 consensus_queue.cc:260] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.161499 20256 ts_tablet_manager.cc:1428] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 15:49:50.163095 20237 heartbeater.cc:503] Master 127.0.97.62:41713 was elected leader, sending a full tablet report...
I20260929 15:49:50.163676 20257 ts_tablet_manager.cc:1428] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Time spent starting tablet: real 0.019s user 0.016s sys 0.001s
I20260929 15:49:50.164894 20258 raft_consensus.cc:348] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.165463 20258 raft_consensus.cc:374] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:50.165593 20258 raft_consensus.cc:729] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ff71b312f2484593e114af3ed3f4d1, State: Initialized, Role: FOLLOWER
I20260929 15:49:50.166318 20258 consensus_queue.cc:260] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.167774 20103 heartbeater.cc:503] Master 127.0.97.62:41713 was elected leader, sending a full tablet report...
I20260929 15:49:50.168759 20258 ts_tablet_manager.cc:1428] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Time spent starting tablet: real 0.017s user 0.008s sys 0.011s
W20260929 15:49:50.190356 20104 tablet.cc:2369] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:50.200479 20262 raft_consensus.cc:482] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:50.200775 20262 raft_consensus.cc:504] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.202443 20262 leader_election.cc:288] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 200397de790240ffb1311b46ea8d8c59 (127.0.97.1:37663), d5ff71b312f2484593e114af3ed3f4d1 (127.0.97.2:35779)
I20260929 15:49:50.209057 20263 raft_consensus.cc:482] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:50.209414 20263 raft_consensus.cc:504] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.210654 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.202924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37663 (local address 127.0.97.3:53227)
0929 15:49:50.203149 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.203171 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.203322 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.203806 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.203811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.203840 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.204057 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.204068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.205406 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.205410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.207563 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.207581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.209806 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.209812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.210028 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.210065 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.210189 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.210329 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 15:49:50.211354 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.202892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53227 (local address 127.0.97.1:37663)
0929 15:49:50.203426 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.203435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.203465 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:49:50.203562 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.203571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.203658 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.203767 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.204255 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.205238 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.207768 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.209291 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.209456 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.209683 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.210948 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.211113 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260929 15:49:50.211730 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.203145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52099 (local address 127.0.97.2:35779)
0929 15:49:50.203436 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.203444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.204020 (+ 576us) server_negotiation.cc:407] Connection header received
0929 15:49:50.204259 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.204265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.204350 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.204474 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.204917 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.205766 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.208668 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.209938 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.210111 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.210201 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.211142 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.211299 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 15:49:50.211753 20263 leader_election.cc:288] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c35aedb5364eb8a5bba11d085cb5eb (127.0.97.3:35595), d5ff71b312f2484593e114af3ed3f4d1 (127.0.97.2:35779)
I20260929 15:49:50.212723 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.202994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35779 (local address 127.0.97.3:52099)
0929 15:49:50.203900 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.203917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.204037 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.204480 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.204482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.204501 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.204760 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.204766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.205935 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.205941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.208331 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.208345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.210528 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.210535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.210921 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.210956 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.212110 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.212338 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":552,"threads_started":1}
I20260929 15:49:50.212749 19924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc28256155c412886630ed048c98279" candidate_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200397de790240ffb1311b46ea8d8c59" is_pre_election: true
I20260929 15:49:50.213205 19924 raft_consensus.cc:2454] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c35aedb5364eb8a5bba11d085cb5eb in term 0.
I20260929 15:49:50.214177 20058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc28256155c412886630ed048c98279" candidate_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ff71b312f2484593e114af3ed3f4d1" is_pre_election: true
I20260929 15:49:50.214879 20058 raft_consensus.cc:2454] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c35aedb5364eb8a5bba11d085cb5eb in term 0.
I20260929 15:49:50.216396 20128 leader_election.cc:302] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c35aedb5364eb8a5bba11d085cb5eb, d5ff71b312f2484593e114af3ed3f4d1; no voters:
I20260929 15:49:50.217691 20262 raft_consensus.cc:2781] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:49:50.217864 20262 raft_consensus.cc:482] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:49:50.217981 20262 raft_consensus.cc:3037] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:50.221571 20262 raft_consensus.cc:504] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.222925 20058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc28256155c412886630ed048c98279" candidate_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ff71b312f2484593e114af3ed3f4d1"
I20260929 15:49:50.223177 20058 raft_consensus.cc:3037] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:50.224076 19924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc28256155c412886630ed048c98279" candidate_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "200397de790240ffb1311b46ea8d8c59"
I20260929 15:49:50.224334 19924 raft_consensus.cc:3037] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:50.226826 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.213746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36297 (local address 127.0.97.3:35595)
0929 15:49:50.215597 (+ 1851us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.215606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.217754 (+ 2148us) server_negotiation.cc:407] Connection header received
0929 15:49:50.218288 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.218296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.218384 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.218455 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.219313 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.220542 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.223455 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.225164 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.225285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.225399 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.226277 (+ 878us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.226451 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1640}
I20260929 15:49:50.227273 20058 raft_consensus.cc:2454] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c35aedb5364eb8a5bba11d085cb5eb in term 1.
I20260929 15:49:50.227748 19924 raft_consensus.cc:2454] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c35aedb5364eb8a5bba11d085cb5eb in term 1.
I20260929 15:49:50.227870 20128 leader_election.cc:302] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c35aedb5364eb8a5bba11d085cb5eb, d5ff71b312f2484593e114af3ed3f4d1; no voters:
I20260929 15:49:50.228045 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.212050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35595 (local address 127.0.97.1:36297)
0929 15:49:50.217641 (+ 5591us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.217665 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.218097 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.218696 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.218700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.218732 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.219143 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.219154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.220722 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.220728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.223216 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.223233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.225819 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.225827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.226038 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.226094 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.227231 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.227470 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5439,"thread_start_us":10931,"threads_started":1}
I20260929 15:49:50.229156 20262 leader_election.cc:288] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [CANDIDATE]: Term 1 election: Requested vote from peers 200397de790240ffb1311b46ea8d8c59 (127.0.97.1:37663), d5ff71b312f2484593e114af3ed3f4d1 (127.0.97.2:35779)
I20260929 15:49:50.229703 20262 raft_consensus.cc:2781] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:50.229350 20192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc28256155c412886630ed048c98279" candidate_uuid: "200397de790240ffb1311b46ea8d8c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" is_pre_election: true
I20260929 15:49:50.230085 20262 raft_consensus.cc:686] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 1 LEADER]: Becoming Leader. State: Replica: b0c35aedb5364eb8a5bba11d085cb5eb, State: Running, Role: LEADER
I20260929 15:49:50.230782 20262 consensus_queue.cc:237] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } }
I20260929 15:49:50.236788 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.212993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32893 (local address 127.0.97.2:35779)
0929 15:49:50.213184 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.213189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.222981 (+ 9792us) server_negotiation.cc:407] Connection header received
0929 15:49:50.228549 (+ 5568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.228555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.228653 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.228712 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:50.230803 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.231963 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.234501 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.235732 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.235887 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.236025 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.236415 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.236562 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:49:50.236891 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.212413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35779 (local address 127.0.97.1:32893)
0929 15:49:50.222865 (+ 10452us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:50.222886 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:50.228363 (+ 5477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:50.228901 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:50.228906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:50.230175 (+ 1269us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:50.230606 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.230620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.232422 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.232428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.234241 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:50.234260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.236007 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.236014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:50.236221 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.236258 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:49:50.236367 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:49:50.236516 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10308,"thread_start_us":4882,"threads_started":1}
I20260929 15:49:50.236984 19780 catalog_manager.cc:5495] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c35aedb5364eb8a5bba11d085cb5eb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c35aedb5364eb8a5bba11d085cb5eb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 } health_report { overall_health: UNKNOWN } } }
I20260929 15:49:50.237550 20058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc28256155c412886630ed048c98279" candidate_uuid: "200397de790240ffb1311b46ea8d8c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ff71b312f2484593e114af3ed3f4d1" is_pre_election: true
I20260929 15:49:50.237915 20058 raft_consensus.cc:2379] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 200397de790240ffb1311b46ea8d8c59 in current term 1: Already voted for candidate b0c35aedb5364eb8a5bba11d085cb5eb in this term.
I20260929 15:49:50.238615 19860 leader_election.cc:302] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 200397de790240ffb1311b46ea8d8c59; no voters: b0c35aedb5364eb8a5bba11d085cb5eb, d5ff71b312f2484593e114af3ed3f4d1
I20260929 15:49:50.239076 20263 raft_consensus.cc:2726] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 15:49:50.262656 20238 tablet.cc:2369] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:50.274526 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.269132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.0.97.3:35595)
0929 15:49:50.269326 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.269332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.269354 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:49:50.269533 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.269537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.269653 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.269813 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:49:50.270178 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.270939 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.272336 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.272734 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.272875 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.273376 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:49:50.273904 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:49:50.274069 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.274128 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.274213 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:49:50.283762 20058 raft_consensus.cc:1261] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 1 FOLLOWER]: Refusing update from remote peer b0c35aedb5364eb8a5bba11d085cb5eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:50.283803 19924 raft_consensus.cc:1261] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 1 FOLLOWER]: Refusing update from remote peer b0c35aedb5364eb8a5bba11d085cb5eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:50.284788 20262 consensus_queue.cc:1035] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ff71b312f2484593e114af3ed3f4d1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:50.285181 20268 consensus_queue.cc:1035] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "200397de790240ffb1311b46ea8d8c59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:50.299666 20273 mvcc.cc:205] Tried to move back new op lower bound from 7334694872190136320 to 7334694871990878208. Current Snapshot: MvccSnapshot[applied={T|T < 7334694872190136320}]
I20260929 15:49:50.316195 19779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:50.316524 19779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:50.320701 19779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d5fca0514e840009cc64f56816eb7db: Sending DeleteTablet for 3 replicas of tablet 9bc28256155c412886630ed048c98279
I20260929 15:49:50.322222 19904 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc28256155c412886630ed048c98279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48540
I20260929 15:49:50.322307 20172 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc28256155c412886630ed048c98279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45262
I20260929 15:49:50.322566 20038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc28256155c412886630ed048c98279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260929 15:49:50.323266 20281 tablet_replica.cc:330] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: stopping tablet replica
I20260929 15:49:50.324641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19841
I20260929 15:49:50.331641 20281 raft_consensus.cc:2229] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:50.332973 20281 raft_consensus.cc:2258] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:50.335676 20282 tablet_replica.cc:330] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: stopping tablet replica
I20260929 15:49:50.336129 20282 raft_consensus.cc:2229] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:49:50.336364 20281 ts_tablet_manager.cc:1905] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:50.336850 20282 raft_consensus.cc:2258] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:50.339587 20282 ts_tablet_manager.cc:1905] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:49:50 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 15:49:50.344974 20283 tablet_replica.cc:330] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: stopping tablet replica
I20260929 15:49:50.345000 20281 ts_tablet_manager.cc:1918] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:50.345191 20281 log.cc:1196] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/9bc28256155c412886630ed048c98279
I20260929 15:49:50.345532 20283 raft_consensus.cc:2229] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:50.345731 20281 ts_tablet_manager.cc:1939] T 9bc28256155c412886630ed048c98279 P 200397de790240ffb1311b46ea8d8c59: Deleting consensus metadata
I20260929 15:49:50.346115 20283 raft_consensus.cc:2258] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:50.347882 19766 catalog_manager.cc:4841] TS 200397de790240ffb1311b46ea8d8c59 (127.0.97.1:37663): tablet 9bc28256155c412886630ed048c98279 (table test-table [id=0c6e80d8a401423e98516433d28c7b20]) successfully deleted
I20260929 15:49:50.350549 20283 ts_tablet_manager.cc:1905] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:50.351047 20282 ts_tablet_manager.cc:1918] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:50.351243 20282 log.cc:1196] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/9bc28256155c412886630ed048c98279
I20260929 15:49:50.351817 20282 ts_tablet_manager.cc:1939] T 9bc28256155c412886630ed048c98279 P b0c35aedb5364eb8a5bba11d085cb5eb: Deleting consensus metadata
I20260929 15:49:50.353856 19764 catalog_manager.cc:4841] TS b0c35aedb5364eb8a5bba11d085cb5eb (127.0.97.3:35595): tablet 9bc28256155c412886630ed048c98279 (table test-table [id=0c6e80d8a401423e98516433d28c7b20]) successfully deleted
I20260929 15:49:50.355567 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.326228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.97.1:37663)
0929 15:49:50.326415 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.326422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.331764 (+ 5342us) server_negotiation.cc:407] Connection header received
0929 15:49:50.332226 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.332234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.332349 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.332469 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:50.335308 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.336449 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.338791 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.339683 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.339824 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.346498 (+ 6674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:50.346517 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:50.346536 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:50.346593 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:50.351090 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.352223 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.352235 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.352239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.352421 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.352925 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.352932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.352938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.353442 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:50.353734 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.354920 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.355177 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:49:50.356232 19964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:50.358604 20283 ts_tablet_manager.cc:1918] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:50.358803 20283 log.cc:1196] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/9bc28256155c412886630ed048c98279
I20260929 15:49:50.359416 20283 ts_tablet_manager.cc:1939] T 9bc28256155c412886630ed048c98279 P d5ff71b312f2484593e114af3ed3f4d1: Deleting consensus metadata
I20260929 15:49:50.361496 19767 catalog_manager.cc:4841] TS d5ff71b312f2484593e114af3ed3f4d1 (127.0.97.2:35779): tablet 9bc28256155c412886630ed048c98279 (table test-table [id=0c6e80d8a401423e98516433d28c7b20]) successfully deleted
I20260929 15:49:50.460980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19841
I20260929 15:49:50.481324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19975
Sep 29 15:49:50 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 15:49:50.499053 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.482200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.97.2:35779)
0929 15:49:50.482371 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.482375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.482393 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:49:50.482505 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.482509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.482579 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.482665 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:50.484299 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.485042 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.487054 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.487504 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.487623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.492958 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:50.492975 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:50.492997 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:50.493038 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:50.495964 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.496892 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.496906 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.496909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.497048 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.497466 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.497470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.497472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.497842 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:50.498175 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.498471 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.498711 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:49:50.499753 20098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:50.592842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19975
I20260929 15:49:50.610405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20109
Sep 29 15:49:50 dist-test-slave-7491 krb5kdc[19735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 15:49:50.627490 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:50.611237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.97.3:35595)
0929 15:49:50.611435 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:50.611442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:50.611461 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:49:50.611690 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:50.611694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:50.611766 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:50.611889 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:50.613372 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.614140 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.615617 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:50.616068 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:50.616251 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:50.620699 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:50.620715 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:50.620731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:50.620782 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:50.624151 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.625221 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.625229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.625232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.625374 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:50.625792 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:50.625797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:50.625803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:50.626243 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:50.626586 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:50.626884 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 15:49:50.627158 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:49:50.628383 20232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:50.757148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20109
I20260929 15:49:50.778318 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 19748
I20260929 15:49:50.779428 19810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:50.892086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 19748
2026-09-29T15:49:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4523 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T15:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:50Z Disabled control of system clock
I20260929 15:49:50.976306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33431
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:49:51.316082 20292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:51.316443 20292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:51.316622 20292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:51.324151 20292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:51.324339 20292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:51.324409 20292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:51.324505 20292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:51.324570 20292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:51.334234 20292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:51.335685 20292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:51.337678 20292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:51.350232 20297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:51.350265 20300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:51.350868 20298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:51.351356 20292 server_base.cc:1034] running on GCE node
I20260929 15:49:51.352300 20292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:51.353888 20292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:51.355131 20292 hybrid_clock.cc:648] HybridClock initialized: now 1790696991355068 us; error 67 us; skew 500 ppm
I20260929 15:49:51.355500 20292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:51.358096 20292 webserver.cc:460] Webserver started at http://127.0.97.62:37185/ using document root <none> and password file <none>
I20260929 15:49:51.358843 20292 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:51.358986 20292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:51.359310 20292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:51.362977 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "4194af8b6a6942919e0d78e21f197579"
format_stamp: "Formatted at 2026-09-29 15:49:51 on dist-test-slave-7491"
server_key: "dbf608a5bef5ff25cdaf9f94e5527ec6"
server_key_iv: "460893404ad02bec1127531e45c9f1fc"
server_key_version: "encryptionkey@0"
I20260929 15:49:51.363873 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "4194af8b6a6942919e0d78e21f197579"
format_stamp: "Formatted at 2026-09-29 15:49:51 on dist-test-slave-7491"
server_key: "dbf608a5bef5ff25cdaf9f94e5527ec6"
server_key_iv: "460893404ad02bec1127531e45c9f1fc"
server_key_version: "encryptionkey@0"
I20260929 15:49:51.369563 20292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:49:51.373445 20306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:51.374925 20292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:49:51.375136 20292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "4194af8b6a6942919e0d78e21f197579"
format_stamp: "Formatted at 2026-09-29 15:49:51 on dist-test-slave-7491"
server_key: "dbf608a5bef5ff25cdaf9f94e5527ec6"
server_key_iv: "460893404ad02bec1127531e45c9f1fc"
server_key_version: "encryptionkey@0"
I20260929 15:49:51.375372 20292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:51.400794 20292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:51.401737 20292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:51.402061 20292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:51.422111 20292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33431
I20260929 15:49:51.422148 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33431 every 8 connection(s)
I20260929 15:49:51.423583 20292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:51.427572 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:51.432915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20292
I20260929 15:49:51.433130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:51.433614 388 external_mini_cluster.cc:1428] Setting key f1dc228f94dfd50fe785b5becf7854ec
I20260929 15:49:51.439561 20358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579: Bootstrap starting.
I20260929 15:49:51.443481 20358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:51.444831 20358 log.cc:824] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:51.448722 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:51.435715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.97.62:33431)
0929 15:49:51.436955 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:51.436966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:51.437015 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:49:51.438789 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:51.438814 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:51.439317 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:51.439671 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:51.440303 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:51.441646 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:51.443684 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:51.444530 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:51.444690 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:51.446029 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:51.446052 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:51.446076 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:51.446806 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:51.446891 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:51.446914 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:51.447007 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:51.447267 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 15:49:51.447643 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":505,"threads_started":1}
I20260929 15:49:51.448776 20358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579: No bootstrap required, opened a new log
I20260929 15:49:51.457436 20358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4194af8b6a6942919e0d78e21f197579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33431 } }
I20260929 15:49:51.458036 20358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:51.458165 20358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4194af8b6a6942919e0d78e21f197579, State: Initialized, Role: FOLLOWER
I20260929 15:49:51.458923 20358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4194af8b6a6942919e0d78e21f197579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33431 } }
I20260929 15:49:51.459163 20358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:51.459271 20358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:51.459406 20358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:51.462107 20358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4194af8b6a6942919e0d78e21f197579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33431 } }
I20260929 15:49:51.462656 20358 leader_election.cc:302] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4194af8b6a6942919e0d78e21f197579; no voters:
I20260929 15:49:51.463255 20358 leader_election.cc:288] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:51.463487 20363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:51.464352 20363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [term 1 LEADER]: Becoming Leader. State: Replica: 4194af8b6a6942919e0d78e21f197579, State: Running, Role: LEADER
I20260929 15:49:51.464932 20363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4194af8b6a6942919e0d78e21f197579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33431 } }
I20260929 15:49:51.465813 20358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:51.467780 20365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4194af8b6a6942919e0d78e21f197579. Latest consensus state: current_term: 1 leader_uuid: "4194af8b6a6942919e0d78e21f197579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4194af8b6a6942919e0d78e21f197579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33431 } } }
I20260929 15:49:51.468268 20365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:51.470757 20364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4194af8b6a6942919e0d78e21f197579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4194af8b6a6942919e0d78e21f197579" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33431 } } }
I20260929 15:49:51.471446 20364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:51.472311 20370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:51.477710 20370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:51.487874 20370 catalog_manager.cc:1347] Generated new cluster ID: aab36f6e912c4655ac7bde10f92413a1
I20260929 15:49:51.488042 20370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:51.501792 20370 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:51.503822 20370 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:51.518575 20370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579: Generated new TSK 0
I20260929 15:49:51.519821 20370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:49:51.530107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33431
--builtin_ntp_servers=127.0.97.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 15:49:51.885187 20382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:51.885511 20382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:51.885646 20382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:51.893374 20382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:51.893501 20382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:51.893679 20382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 15:49:51.903438 20382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:51.904894 20382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:51.906764 20382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:51.921805 20390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:51.921860 20388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:51.922153 20387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:51.923818 20382 server_base.cc:1034] running on GCE node
I20260929 15:49:51.924633 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:51.925782 20382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:51.927073 20382 hybrid_clock.cc:648] HybridClock initialized: now 1790696991927014 us; error 46 us; skew 500 ppm
I20260929 15:49:51.927475 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:51.930364 20382 webserver.cc:460] Webserver started at http://127.0.97.1:42761/ using document root <none> and password file <none>
I20260929 15:49:51.931145 20382 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:51.931288 20382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:51.931774 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:51.935045 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data/instance:
uuid: "1fc02fa251c543a1afbb6b1983d954a2"
format_stamp: "Formatted at 2026-09-29 15:49:51 on dist-test-slave-7491"
server_key: "e09240bbd870728849c696e9b7a478b9"
server_key_iv: "710334d11dc6a6906179f559dda993f6"
server_key_version: "encryptionkey@0"
I20260929 15:49:51.935947 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1fc02fa251c543a1afbb6b1983d954a2"
format_stamp: "Formatted at 2026-09-29 15:49:51 on dist-test-slave-7491"
server_key: "e09240bbd870728849c696e9b7a478b9"
server_key_iv: "710334d11dc6a6906179f559dda993f6"
server_key_version: "encryptionkey@0"
I20260929 15:49:51.941759 20382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 15:49:51.945518 20396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:51.947134 20382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:49:51.947299 20382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal
uuid: "1fc02fa251c543a1afbb6b1983d954a2"
format_stamp: "Formatted at 2026-09-29 15:49:51 on dist-test-slave-7491"
server_key: "e09240bbd870728849c696e9b7a478b9"
server_key_iv: "710334d11dc6a6906179f559dda993f6"
server_key_version: "encryptionkey@0"
I20260929 15:49:51.947502 20382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:51.976001 20382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:51.976945 20382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:51.977196 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:51.978328 20382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:51.980410 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:51.980530 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:51.980611 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:51.980674 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.019162 20382 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46555
I20260929 15:49:52.019219 20508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46555 every 8 connection(s)
I20260929 15:49:52.020596 20382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/data/info.pb
I20260929 15:49:52.021361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20382
I20260929 15:49:52.021683 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal/instance
I20260929 15:49:52.022279 388 external_mini_cluster.cc:1428] Setting key cab86a91f25a58a263ecbcc39d8e5293
I20260929 15:49:52.025619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33431
--builtin_ntp_servers=127.0.97.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:49:52.039165 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:52.023665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60705 (local address 127.0.97.62:33431)
0929 15:49:52.024195 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:52.024202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:52.026143 (+ 1941us) server_negotiation.cc:407] Connection header received
0929 15:49:52.029505 (+ 3362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:52.029510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:52.029707 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:52.029826 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:52.031838 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.032838 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.035768 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.036294 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.036453 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:52.038174 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:52.038181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:52.038185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:52.038209 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:52.038261 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:52.038276 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:52.038398 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:52.038666 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 15:49:52.038793 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":264,"threads_started":1}
I20260929 15:49:52.040362 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:52.024457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33431 (local address 127.0.97.1:60705)
0929 15:49:52.025787 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:52.025844 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:52.029062 (+ 3218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:52.030287 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:52.030304 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:52.030477 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:52.031538 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:52.031571 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.033051 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.033059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:52.035525 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:52.035552 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.036581 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.036589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:52.036854 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:52.037664 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:52.037667 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:52.037721 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:52.037825 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:52.038398 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:52.038414 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:52.038425 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:49:52.038650 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 15:49:52.039143 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":471,"threads_started":1}
I20260929 15:49:52.042286 20509 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33431
I20260929 15:49:52.042642 20509 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:52.043473 20509 heartbeater.cc:511] Master 127.0.97.62:33431 requested a full tablet report, sending...
I20260929 15:49:52.046183 20323 ts_manager.cc:194] Registered new tserver with Master: 1fc02fa251c543a1afbb6b1983d954a2 (127.0.97.1:46555)
I20260929 15:49:52.048763 20323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60705
W20260929 15:49:52.385010 20513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:52.385334 20513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:52.385426 20513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:52.393380 20513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:52.393509 20513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:52.393628 20513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 15:49:52.403204 20513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:52.404542 20513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:52.406319 20513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:52.420876 20518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:52.421023 20521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:52.420929 20519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:52.422022 20513 server_base.cc:1034] running on GCE node
I20260929 15:49:52.422763 20513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:52.424072 20513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:52.425352 20513 hybrid_clock.cc:648] HybridClock initialized: now 1790696992425285 us; error 76 us; skew 500 ppm
I20260929 15:49:52.425714 20513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:52.428725 20513 webserver.cc:460] Webserver started at http://127.0.97.2:42509/ using document root <none> and password file <none>
I20260929 15:49:52.429627 20513 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:52.429746 20513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:52.430087 20513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:52.434119 20513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data/instance:
uuid: "f013c6df063b4cec89f346e7a3c9aacb"
format_stamp: "Formatted at 2026-09-29 15:49:52 on dist-test-slave-7491"
server_key: "dbd4d59b1401a898c5a91312e5989993"
server_key_iv: "ce719d5c5ef23462882c7dc31f065185"
server_key_version: "encryptionkey@0"
I20260929 15:49:52.435205 20513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f013c6df063b4cec89f346e7a3c9aacb"
format_stamp: "Formatted at 2026-09-29 15:49:52 on dist-test-slave-7491"
server_key: "dbd4d59b1401a898c5a91312e5989993"
server_key_iv: "ce719d5c5ef23462882c7dc31f065185"
server_key_version: "encryptionkey@0"
I20260929 15:49:52.442293 20513 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:49:52.446919 20527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.448714 20513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:49:52.448944 20513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal
uuid: "f013c6df063b4cec89f346e7a3c9aacb"
format_stamp: "Formatted at 2026-09-29 15:49:52 on dist-test-slave-7491"
server_key: "dbd4d59b1401a898c5a91312e5989993"
server_key_iv: "ce719d5c5ef23462882c7dc31f065185"
server_key_version: "encryptionkey@0"
I20260929 15:49:52.449221 20513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:52.467231 20513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:52.468219 20513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:52.468523 20513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:52.469686 20513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:52.471431 20513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:52.471562 20513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.471706 20513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:52.471797 20513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.507720 20513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43529
I20260929 15:49:52.507750 20639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43529 every 8 connection(s)
I20260929 15:49:52.509208 20513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/data/info.pb
I20260929 15:49:52.519285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20513
I20260929 15:49:52.519426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal/instance
I20260929 15:49:52.519912 388 external_mini_cluster.cc:1428] Setting key f1feffb13e2b82b2ef833938cfb2b3b9
I20260929 15:49:52.523180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33431
--builtin_ntp_servers=127.0.97.20:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 15:49:52.523514 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:52.512281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55421 (local address 127.0.97.62:33431)
0929 15:49:52.512519 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:52.512526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:52.514012 (+ 1486us) server_negotiation.cc:407] Connection header received
0929 15:49:52.515972 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:52.515977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:52.516056 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:52.516159 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:52.517421 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.518173 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.520359 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.520864 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.521032 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:52.522596 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:52.522602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:52.522616 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:52.522640 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:52.522682 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:52.522699 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:52.522797 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:52.523031 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 15:49:52.523146 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:49:52.524642 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:52.512807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33431 (local address 127.0.97.2:55421)
0929 15:49:52.513822 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:52.513861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:52.515602 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:52.516505 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:52.516516 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:52.516616 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:52.517242 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:52.517260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.518351 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.518356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:52.520108 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:52.520126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:52.521148 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:52.521158 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:52.521404 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:52.522210 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:52.522213 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:52.522262 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:52.522351 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:52.522795 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:52.522803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:52.522809 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:49:52.522963 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 15:49:52.523369 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":316,"threads_started":1}
I20260929 15:49:52.526342 20640 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33431
I20260929 15:49:52.526702 20640 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:52.527701 20640 heartbeater.cc:511] Master 127.0.97.62:33431 requested a full tablet report, sending...
I20260929 15:49:52.529500 20323 ts_manager.cc:194] Registered new tserver with Master: f013c6df063b4cec89f346e7a3c9aacb (127.0.97.2:43529)
I20260929 15:49:52.530366 20323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:55421
W20260929 15:49:52.865973 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:52.866276 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:52.866369 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:52.874261 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:52.874452 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:52.874647 20643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 15:49:52.884002 20643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:52.885358 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:52.887118 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:52.902267 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:52.902339 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:52.902444 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:52.903370 20643 server_base.cc:1034] running on GCE node
I20260929 15:49:52.904124 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:52.905318 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:52.906581 20643 hybrid_clock.cc:648] HybridClock initialized: now 1790696992906483 us; error 84 us; skew 500 ppm
I20260929 15:49:52.906932 20643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:49:52.909726 20643 webserver.cc:460] Webserver started at http://127.0.97.3:45525/ using document root <none> and password file <none>
I20260929 15:49:52.910528 20643 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:52.910627 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:52.910904 20643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:52.914022 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data/instance:
uuid: "adc6b8d0e27248c3834db32623d4c3e6"
format_stamp: "Formatted at 2026-09-29 15:49:52 on dist-test-slave-7491"
server_key: "9c188a65c92fe215275d5ffdec9a0810"
server_key_iv: "6f1c81451b7f5bd96c0e9d0cc71fa189"
server_key_version: "encryptionkey@0"
I20260929 15:49:52.914849 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal/instance:
uuid: "adc6b8d0e27248c3834db32623d4c3e6"
format_stamp: "Formatted at 2026-09-29 15:49:52 on dist-test-slave-7491"
server_key: "9c188a65c92fe215275d5ffdec9a0810"
server_key_iv: "6f1c81451b7f5bd96c0e9d0cc71fa189"
server_key_version: "encryptionkey@0"
I20260929 15:49:52.920994 20643 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:49:52.924836 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.926367 20643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 15:49:52.926580 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal
uuid: "adc6b8d0e27248c3834db32623d4c3e6"
format_stamp: "Formatted at 2026-09-29 15:49:52 on dist-test-slave-7491"
server_key: "9c188a65c92fe215275d5ffdec9a0810"
server_key_iv: "6f1c81451b7f5bd96c0e9d0cc71fa189"
server_key_version: "encryptionkey@0"
I20260929 15:49:52.926841 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:52.949384 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:52.950588 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:52.950964 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:52.952078 20643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:49:52.954975 20643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:49:52.955117 20643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.955255 20643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:49:52.955338 20643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:52.993569 20643 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36305
I20260929 15:49:52.993664 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36305 every 8 connection(s)
I20260929 15:49:52.994882 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/data/info.pb
I20260929 15:49:53.002048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20643
I20260929 15:49:53.002314 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal/instance
I20260929 15:49:53.002849 388 external_mini_cluster.cc:1428] Setting key b632a04fe305c83f0d7775d7c6b0223a
I20260929 15:49:53.009099 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:52.997648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59013 (local address 127.0.97.62:33431)
0929 15:49:52.997859 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:52.997864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:52.999226 (+ 1362us) server_negotiation.cc:407] Connection header received
0929 15:49:53.001198 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.001204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.001282 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.001381 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:53.002635 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.003685 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.005889 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.006411 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.006640 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.008215 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:53.008226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:53.008236 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:53.008257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:53.008291 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:53.008305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:53.008391 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.008698 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.008792 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:49:53.010139 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:52.998097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33431 (local address 127.0.97.3:59013)
0929 15:49:52.999055 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:52.999094 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:53.000861 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:53.001600 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:53.001609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:53.001716 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:49:53.002408 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.002434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.003887 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.003893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.005700 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.005717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.006757 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.006763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.006981 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.007753 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:49:53.007756 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:49:53.007802 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:49:53.007898 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:49:53.008427 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:49:53.008436 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:49:53.008443 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:49:53.008638 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 15:49:53.008978 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":329,"threads_started":1}
I20260929 15:49:53.011976 20770 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33431
I20260929 15:49:53.012499 20770 heartbeater.cc:464] Registering TS with master...
I20260929 15:49:53.013322 20770 heartbeater.cc:511] Master 127.0.97.62:33431 requested a full tablet report, sending...
I20260929 15:49:53.014984 20323 ts_manager.cc:194] Registered new tserver with Master: adc6b8d0e27248c3834db32623d4c3e6 (127.0.97.3:36305)
I20260929 15:49:53.015967 20323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59013
I20260929 15:49:53.019001 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:49:53.029762 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.023519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.0.97.62:33431)
0929 15:49:53.023742 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.023750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.024126 (+ 376us) server_negotiation.cc:407] Connection header received
0929 15:49:53.024370 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.024375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.024443 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.024553 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:53.025104 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.025792 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.027218 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.027687 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.027838 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.029018 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:53.029024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:53.029027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:53.029047 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:53.029079 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:53.029091 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:53.029191 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.029333 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.029438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:49:53.034345 20323 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:49:53.041469 20323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:49:53.053126 20509 heartbeater.cc:503] Master 127.0.97.62:33431 was elected leader, sending a full tablet report...
I20260929 15:49:53.084272 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.055403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43529 (local address 127.0.0.1:35810)
0929 15:49:53.068352 (+ 12949us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:53.068432 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:53.068747 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:53.073180 (+ 4433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:53.073191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:53.073238 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:53.073727 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.073745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.076834 (+ 3089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.076844 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.080721 (+ 3877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.080741 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.083166 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.083175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.083446 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.083512 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 15:49:53.083657 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 15:49:53.083867 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12468,"thread_start_us":15773,"threads_started":1}
I20260929 15:49:53.085465 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.055544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.0.97.2:43529)
0929 15:49:53.071908 (+ 16364us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.071923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.071966 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:49:53.072106 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.072118 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.072508 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.072778 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:53.074005 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.076582 (+ 2577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.081069 (+ 4487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.082641 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.082772 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.083044 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.084870 (+ 1826us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.085078 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16127,"thread_start_us":16224,"threads_started":1}
I20260929 15:49:53.088114 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.056065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.0.97.1:46555)
0929 15:49:53.079930 (+ 23865us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.079939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.079971 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:49:53.080082 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.080088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.080422 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.080632 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:53.081387 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.082905 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.085591 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.086983 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.087159 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.087449 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.087603 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.087787 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23687,"thread_start_us":23558,"threads_started":1}
I20260929 15:49:53.088092 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.055557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46555 (local address 127.0.0.1:53508)
0929 15:49:53.071517 (+ 15960us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:53.071543 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:53.071702 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:53.080638 (+ 8936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:53.080645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:53.080687 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:53.081163 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.081180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.083069 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.083075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.085278 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.085295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.087207 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.087213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.087396 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.087441 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 15:49:53.087550 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:49:53.087716 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15792,"thread_start_us":11752,"threads_started":1}
I20260929 15:49:53.089298 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.055988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36305 (local address 127.0.0.1:47420)
0929 15:49:53.081183 (+ 25195us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:53.081199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:53.081362 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:53.082299 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:53.082303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:53.082332 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:53.082743 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.082755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.084290 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.084295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.085997 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.086010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.088101 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.088109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.088650 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.088683 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:49:53.088776 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:49:53.088923 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25064,"mutex_wait_us":4,"thread_start_us":25506,"threads_started":1}
I20260929 15:49:53.090335 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.056564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.97.3:36305)
0929 15:49:53.079423 (+ 22859us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.079437 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.081322 (+ 1885us) server_negotiation.cc:407] Connection header received
0929 15:49:53.081535 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.081543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.081791 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.082003 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:53.082953 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.084092 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.086198 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.087511 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.087690 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.087945 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.089745 (+ 1800us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.089945 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22645,"thread_start_us":22465,"threads_started":1}
I20260929 15:49:53.090193 20575 tablet_service.cc:1467] Processing CreateTablet for tablet 25f9e8ab14ac4aa5a122df4421d3856f (DEFAULT_TABLE table=test-table [id=cfc84b18e0a149c4ad219e66d6def0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:53.091984 20575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9e8ab14ac4aa5a122df4421d3856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:53.091914 20444 tablet_service.cc:1467] Processing CreateTablet for tablet 25f9e8ab14ac4aa5a122df4421d3856f (DEFAULT_TABLE table=test-table [id=cfc84b18e0a149c4ad219e66d6def0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:53.093349 20444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9e8ab14ac4aa5a122df4421d3856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:53.094276 20705 tablet_service.cc:1467] Processing CreateTablet for tablet 25f9e8ab14ac4aa5a122df4421d3856f (DEFAULT_TABLE table=test-table [id=cfc84b18e0a149c4ad219e66d6def0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:49:53.096259 20705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9e8ab14ac4aa5a122df4421d3856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:53.107741 20788 tablet_bootstrap.cc:492] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Bootstrap starting.
I20260929 15:49:53.111575 20788 tablet_bootstrap.cc:654] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:53.112705 20789 tablet_bootstrap.cc:492] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Bootstrap starting.
I20260929 15:49:53.113169 20788 log.cc:824] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:53.118064 20789 tablet_bootstrap.cc:654] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:53.119401 20789 log.cc:824] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:53.119663 20788 tablet_bootstrap.cc:492] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: No bootstrap required, opened a new log
I20260929 15:49:53.119980 20788 ts_tablet_manager.cc:1397] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260929 15:49:53.120455 20790 tablet_bootstrap.cc:492] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Bootstrap starting.
I20260929 15:49:53.122699 20789 tablet_bootstrap.cc:492] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: No bootstrap required, opened a new log
I20260929 15:49:53.123020 20789 ts_tablet_manager.cc:1397] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260929 15:49:53.124233 20790 tablet_bootstrap.cc:654] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:53.125519 20790 log.cc:824] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:53.131448 20790 tablet_bootstrap.cc:492] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: No bootstrap required, opened a new log
I20260929 15:49:53.131788 20790 ts_tablet_manager.cc:1397] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260929 15:49:53.131748 20789 raft_consensus.cc:348] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.131727 20788 raft_consensus.cc:348] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.132257 20789 raft_consensus.cc:374] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:53.132299 20788 raft_consensus.cc:374] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:53.132355 20789 raft_consensus.cc:729] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc02fa251c543a1afbb6b1983d954a2, State: Initialized, Role: FOLLOWER
I20260929 15:49:53.132419 20788 raft_consensus.cc:729] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f013c6df063b4cec89f346e7a3c9aacb, State: Initialized, Role: FOLLOWER
I20260929 15:49:53.133052 20789 consensus_queue.cc:260] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.133163 20788 consensus_queue.cc:260] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.134547 20640 heartbeater.cc:503] Master 127.0.97.62:33431 was elected leader, sending a full tablet report...
I20260929 15:49:53.136791 20788 ts_tablet_manager.cc:1428] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260929 15:49:53.138633 20789 ts_tablet_manager.cc:1428] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Time spent starting tablet: real 0.015s user 0.008s sys 0.009s
I20260929 15:49:53.142887 20790 raft_consensus.cc:348] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.143416 20790 raft_consensus.cc:374] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:53.143529 20790 raft_consensus.cc:729] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc6b8d0e27248c3834db32623d4c3e6, State: Initialized, Role: FOLLOWER
I20260929 15:49:53.144148 20790 consensus_queue.cc:260] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.145304 20770 heartbeater.cc:503] Master 127.0.97.62:33431 was elected leader, sending a full tablet report...
I20260929 15:49:53.146005 20790 ts_tablet_manager.cc:1428] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260929 15:49:53.246951 20771 tablet.cc:2369] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:49:53.261807 20641 tablet.cc:2369] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:49:53.273428 20510 tablet.cc:2369] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:49:53.381129 20795 raft_consensus.cc:482] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:49:53.381501 20795 raft_consensus.cc:504] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.383244 20795 leader_election.cc:288] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fc02fa251c543a1afbb6b1983d954a2 (127.0.97.1:46555), adc6b8d0e27248c3834db32623d4c3e6 (127.0.97.3:36305)
I20260929 15:49:53.392611 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.383808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38987 (local address 127.0.97.3:36305)
0929 15:49:53.384082 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.384089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.384542 (+ 453us) server_negotiation.cc:407] Connection header received
0929 15:49:53.384776 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.384781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.384865 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.384971 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:53.385792 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.386893 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.390150 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.391423 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.391575 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.391690 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.392206 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.392344 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260929 15:49:53.392625 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.383797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57065 (local address 127.0.97.1:46555)
0929 15:49:53.384043 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.384051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.384645 (+ 594us) server_negotiation.cc:407] Connection header received
0929 15:49:53.384775 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.384782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.384865 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.384972 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:49:53.385793 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.386893 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.390137 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.391423 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.391574 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.391690 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.392206 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.392344 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:49:53.392781 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.383585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36305 (local address 127.0.97.2:38987)
0929 15:49:53.384412 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:53.384433 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:53.384617 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:53.385011 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:53.385015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:53.385100 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:53.385621 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.385629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.387045 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.387049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.389852 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.389871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.391743 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.391756 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.391977 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.392016 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:49:53.392126 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 15:49:53.392338 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":227,"spinlock_wait_cycles":24960,"thread_start_us":416,"threads_started":1}
I20260929 15:49:53.393414 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.383664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46555 (local address 127.0.97.2:57065)
0929 15:49:53.384543 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 15:49:53.384558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:49:53.384654 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:49:53.385011 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:49:53.385023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:49:53.385042 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:49:53.385612 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.385629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.387043 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.387049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.389854 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:49:53.389871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.391766 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.391772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:49:53.391983 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.392024 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:49:53.392126 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:49:53.392338 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":332,"thread_start_us":568,"threads_started":1}
I20260929 15:49:53.393996 20725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9e8ab14ac4aa5a122df4421d3856f" candidate_uuid: "f013c6df063b4cec89f346e7a3c9aacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc6b8d0e27248c3834db32623d4c3e6" is_pre_election: true
I20260929 15:49:53.394529 20725 raft_consensus.cc:2454] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f013c6df063b4cec89f346e7a3c9aacb in term 0.
I20260929 15:49:53.394630 20464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9e8ab14ac4aa5a122df4421d3856f" candidate_uuid: "f013c6df063b4cec89f346e7a3c9aacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc02fa251c543a1afbb6b1983d954a2" is_pre_election: true
I20260929 15:49:53.395159 20464 raft_consensus.cc:2454] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f013c6df063b4cec89f346e7a3c9aacb in term 0.
I20260929 15:49:53.395278 20530 leader_election.cc:302] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc6b8d0e27248c3834db32623d4c3e6, f013c6df063b4cec89f346e7a3c9aacb; no voters:
I20260929 15:49:53.395766 20795 raft_consensus.cc:2781] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:49:53.395929 20795 raft_consensus.cc:482] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:49:53.396044 20795 raft_consensus.cc:3037] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:53.398353 20795 raft_consensus.cc:504] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.399017 20795 leader_election.cc:288] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [CANDIDATE]: Term 1 election: Requested vote from peers 1fc02fa251c543a1afbb6b1983d954a2 (127.0.97.1:46555), adc6b8d0e27248c3834db32623d4c3e6 (127.0.97.3:36305)
I20260929 15:49:53.399667 20464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9e8ab14ac4aa5a122df4421d3856f" candidate_uuid: "f013c6df063b4cec89f346e7a3c9aacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc02fa251c543a1afbb6b1983d954a2"
I20260929 15:49:53.399667 20725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9e8ab14ac4aa5a122df4421d3856f" candidate_uuid: "f013c6df063b4cec89f346e7a3c9aacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc6b8d0e27248c3834db32623d4c3e6"
I20260929 15:49:53.399901 20725 raft_consensus.cc:3037] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:53.399900 20464 raft_consensus.cc:3037] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:53.403318 20725 raft_consensus.cc:2454] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f013c6df063b4cec89f346e7a3c9aacb in term 1.
I20260929 15:49:53.403318 20464 raft_consensus.cc:2454] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f013c6df063b4cec89f346e7a3c9aacb in term 1.
I20260929 15:49:53.403908 20531 leader_election.cc:302] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fc02fa251c543a1afbb6b1983d954a2, adc6b8d0e27248c3834db32623d4c3e6, f013c6df063b4cec89f346e7a3c9aacb; no voters:
I20260929 15:49:53.404314 20795 raft_consensus.cc:2781] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:53.404783 20795 raft_consensus.cc:686] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 1 LEADER]: Becoming Leader. State: Replica: f013c6df063b4cec89f346e7a3c9aacb, State: Running, Role: LEADER
I20260929 15:49:53.405306 20795 consensus_queue.cc:237] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } }
I20260929 15:49:53.410319 20322 catalog_manager.cc:5495] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb reported cstate change: term changed from 0 to 1, leader changed from <none> to f013c6df063b4cec89f346e7a3c9aacb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f013c6df063b4cec89f346e7a3c9aacb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f013c6df063b4cec89f346e7a3c9aacb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20260929 15:49:53.457353 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.451605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.0.97.2:43529)
0929 15:49:53.451916 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.451924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.451952 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:49:53.452276 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.452283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.452431 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.452604 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:49:53.452927 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.453797 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.455218 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.455613 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.455713 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.456229 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:49:53.456776 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:49:53.456887 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.457038 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.457106 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 15:49:53.466053 20725 raft_consensus.cc:1261] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 1 FOLLOWER]: Refusing update from remote peer f013c6df063b4cec89f346e7a3c9aacb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:53.466053 20464 raft_consensus.cc:1261] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 1 FOLLOWER]: Refusing update from remote peer f013c6df063b4cec89f346e7a3c9aacb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:49:53.467233 20795 consensus_queue.cc:1035] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc6b8d0e27248c3834db32623d4c3e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:53.467576 20799 consensus_queue.cc:1035] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc02fa251c543a1afbb6b1983d954a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:49:53.485905 20804 mvcc.cc:205] Tried to move back new op lower bound from 7334694885225865216 to 7334694884992049152. Current Snapshot: MvccSnapshot[applied={T|T < 7334694885225865216}]
I20260929 15:49:53.494386 20322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:53.494709 20322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:49:53.498764 20322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4194af8b6a6942919e0d78e21f197579: Sending DeleteTablet for 3 replicas of tablet 25f9e8ab14ac4aa5a122df4421d3856f
I20260929 15:49:53.500141 20444 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f9e8ab14ac4aa5a122df4421d3856f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:53 UTC) from {username='slave'} at 127.0.0.1:53508
I20260929 15:49:53.500532 20705 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f9e8ab14ac4aa5a122df4421d3856f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:53 UTC) from {username='slave'} at 127.0.0.1:47420
I20260929 15:49:53.500933 20575 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f9e8ab14ac4aa5a122df4421d3856f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:49:53 UTC) from {username='slave'} at 127.0.0.1:35810
I20260929 15:49:53.501384 20811 tablet_replica.cc:330] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: stopping tablet replica
I20260929 15:49:53.502288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20382
I20260929 15:49:53.502617 20811 raft_consensus.cc:2229] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:53.503378 20811 raft_consensus.cc:2258] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:53.504719 20813 tablet_replica.cc:330] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: stopping tablet replica
I20260929 15:49:53.505404 20813 raft_consensus.cc:2229] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:49:53.506409 20813 raft_consensus.cc:2258] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:53.507452 20812 tablet_replica.cc:330] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: stopping tablet replica
I20260929 15:49:53.507862 20812 raft_consensus.cc:2229] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:49:53.508459 20812 raft_consensus.cc:2258] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:49:53.511154 20813 ts_tablet_manager.cc:1905] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:53.511152 20811 ts_tablet_manager.cc:1905] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:53.515427 20812 ts_tablet_manager.cc:1905] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:49:53.522493 20812 ts_tablet_manager.cc:1918] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:53.522696 20812 log.cc:1196] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-2/wal/wals/25f9e8ab14ac4aa5a122df4421d3856f
I20260929 15:49:53.523242 20812 ts_tablet_manager.cc:1939] T 25f9e8ab14ac4aa5a122df4421d3856f P adc6b8d0e27248c3834db32623d4c3e6: Deleting consensus metadata
I20260929 15:49:53.523325 20811 ts_tablet_manager.cc:1918] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:53.523536 20811 log.cc:1196] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-0/wal/wals/25f9e8ab14ac4aa5a122df4421d3856f
I20260929 15:49:53.524099 20811 ts_tablet_manager.cc:1939] T 25f9e8ab14ac4aa5a122df4421d3856f P 1fc02fa251c543a1afbb6b1983d954a2: Deleting consensus metadata
I20260929 15:49:53.525334 20309 catalog_manager.cc:4841] TS adc6b8d0e27248c3834db32623d4c3e6 (127.0.97.3:36305): tablet 25f9e8ab14ac4aa5a122df4421d3856f (table test-table [id=cfc84b18e0a149c4ad219e66d6def0fb]) successfully deleted
I20260929 15:49:53.525895 20310 catalog_manager.cc:4841] TS 1fc02fa251c543a1afbb6b1983d954a2 (127.0.97.1:46555): tablet 25f9e8ab14ac4aa5a122df4421d3856f (table test-table [id=cfc84b18e0a149c4ad219e66d6def0fb]) successfully deleted
I20260929 15:49:53.526001 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.504626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.97.1:46555)
0929 15:49:53.504816 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.504823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.508877 (+ 4054us) server_negotiation.cc:407] Connection header received
0929 15:49:53.509539 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.509546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.509676 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.509783 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:53.511437 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.512627 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.516807 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.517532 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.517704 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.524004 (+ 6300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:53.524017 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:53.524036 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:53.524868 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:53.524934 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:53.524963 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:53.525065 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.525397 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.525534 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":64}
I20260929 15:49:53.526714 20813 ts_tablet_manager.cc:1918] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:49:53.526975 20813 log.cc:1196] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Deleting WAL directory at /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790696645767654-388-0/minicluster-data/ts-1/wal/wals/25f9e8ab14ac4aa5a122df4421d3856f
I20260929 15:49:53.527169 20504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:53.527734 20813 ts_tablet_manager.cc:1939] T 25f9e8ab14ac4aa5a122df4421d3856f P f013c6df063b4cec89f346e7a3c9aacb: Deleting consensus metadata
I20260929 15:49:53.529404 20308 catalog_manager.cc:4841] TS f013c6df063b4cec89f346e7a3c9aacb (127.0.97.2:43529): tablet 25f9e8ab14ac4aa5a122df4421d3856f (table test-table [id=cfc84b18e0a149c4ad219e66d6def0fb]) successfully deleted
I20260929 15:49:53.659241 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20382
I20260929 15:49:53.679230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20513
I20260929 15:49:53.686545 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.680124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.0.97.2:43529)
0929 15:49:53.680329 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.680335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.680385 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:49:53.680627 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.680632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.680732 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.680864 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:53.681283 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.682085 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.683356 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.683784 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.683929 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.685121 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:53.685133 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:53.685148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:53.685782 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:53.685830 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:53.685849 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:53.685985 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.686096 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.686196 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 15:49:53.687301 20635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:53.802742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20513
I20260929 15:49:53.821676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20643
I20260929 15:49:53.828336 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:53.822416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.97.3:36305)
0929 15:49:53.822586 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:53.822590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:53.822611 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:49:53.822780 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:53.822784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:53.822873 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:53.822961 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:53.823312 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.824017 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.825333 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:53.825738 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:53.825844 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:53.826915 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:53.826933 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:53.826948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:53.827515 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:53.827561 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:53.827591 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:53.827700 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:53.827830 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 15:49:53.827918 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:49:53.829123 20765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:53.918709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20643
I20260929 15:49:53.935280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20292
I20260929 15:49:53.936465 20353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:54.049952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20292
2026-09-29T15:49:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3160 ms)
[----------] 6 tests from EncryptionPolicyTest (23512 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:49:54 dist-test-slave-7491 krb5kdc[20821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:49:54 dist-test-slave-7491 krb5kdc[20821](info): set up 2 sockets
Sep 29 15:49:54 dist-test-slave-7491 krb5kdc[20821](info): commencing operation
krb5kdc: starting...
W20260929 15:49:55.184823 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:49:55 dist-test-slave-7491 krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:49:55.453413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:55.803803 20834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:55.804219 20834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:55.804405 20834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:55.812337 20834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:55.812475 20834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:55.812566 20834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:55.812659 20834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:55.812721 20834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:55.823199 20834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:55.824695 20834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:55.826086 20834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:55.839491 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:55.839648 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:55.839602 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:55.840679 20834 server_base.cc:1034] running on GCE node
I20260929 15:49:55.841670 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:55.843295 20834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:55.844573 20834 hybrid_clock.cc:648] HybridClock initialized: now 1790696995844543 us; error 68 us; skew 500 ppm
I20260929 15:49:55.844995 20834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:55 dist-test-slave-7491 krb5kdc[20821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:55.848572 20834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:49:55.850489 20834 webserver.cc:460] Webserver started at http://127.0.0.1:43459/ using document root <none> and password file <none>
I20260929 15:49:55.851260 20834 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:55.851418 20834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:55.851747 20834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:55.854982 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "9225cb3bd40b4b46b821097c2df4d962"
format_stamp: "Formatted at 2026-09-29 15:49:55 on dist-test-slave-7491"
server_key: "b930c637eb946665028c924b812e8828"
server_key_iv: "d8ae59e83c578f82cde8f3738fae273b"
server_key_version: "encryptionkey@0"
I20260929 15:49:55.855974 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "9225cb3bd40b4b46b821097c2df4d962"
format_stamp: "Formatted at 2026-09-29 15:49:55 on dist-test-slave-7491"
server_key: "b930c637eb946665028c924b812e8828"
server_key_iv: "d8ae59e83c578f82cde8f3738fae273b"
server_key_version: "encryptionkey@0"
I20260929 15:49:55.862094 20834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:49:55.866187 20849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:55.867857 20834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:49:55.868120 20834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "9225cb3bd40b4b46b821097c2df4d962"
format_stamp: "Formatted at 2026-09-29 15:49:55 on dist-test-slave-7491"
server_key: "b930c637eb946665028c924b812e8828"
server_key_iv: "d8ae59e83c578f82cde8f3738fae273b"
server_key_version: "encryptionkey@0"
I20260929 15:49:55.868408 20834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:55.882822 20834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:55.887929 20834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:55.888355 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:55.910725 20834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44083
I20260929 15:49:55.910705 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44083 every 8 connection(s)
I20260929 15:49:55.912273 20834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:55.916430 20901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:55.920769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20834
I20260929 15:49:55.921008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:55.921828 388 external_mini_cluster.cc:1428] Setting key 931aec1dc1be4c4f28a6b861ab04a202
I20260929 15:49:55.930773 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962: Bootstrap starting.
I20260929 15:49:55.934679 20901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:55.935811 20901 log.cc:824] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:49:55 dist-test-slave-7491 krb5kdc[20821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:49:55.938890 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962: No bootstrap required, opened a new log
I20260929 15:49:55.947758 20901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9225cb3bd40b4b46b821097c2df4d962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260929 15:49:55.948415 20901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:55.948552 20901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9225cb3bd40b4b46b821097c2df4d962, State: Initialized, Role: FOLLOWER
I20260929 15:49:55.949178 20901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9225cb3bd40b4b46b821097c2df4d962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260929 15:49:55.949461 20901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:55.949615 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:55.924005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41265 (local address 127.0.0.1:44083)
0929 15:49:55.924849 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:55.924861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:55.924907 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:49:55.926680 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:55.926707 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:55.927221 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:55.927725 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:55.930069 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:55.931534 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:55.933420 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:55.934298 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:55.934564 (+ 266us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:55.940445 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:55.940455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:55.940471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:55.940541 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:55.944300 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:55.945370 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:55.945379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:55.945386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:55.945579 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:55.946105 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:55.946109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:55.946111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:55.946786 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:55.947202 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:55.947742 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 15:49:55.948105 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":384,"threads_started":1}
I20260929 15:49:55.949645 20901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:55.949884 20901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:55.953154 20901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9225cb3bd40b4b46b821097c2df4d962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260929 15:49:55.953786 20901 leader_election.cc:302] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9225cb3bd40b4b46b821097c2df4d962; no voters:
I20260929 15:49:55.954514 20901 leader_election.cc:288] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:55.954746 20906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:55.955364 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:49:55.955814 20906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [term 1 LEADER]: Becoming Leader. State: Replica: 9225cb3bd40b4b46b821097c2df4d962, State: Running, Role: LEADER
I20260929 15:49:55.956583 20906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9225cb3bd40b4b46b821097c2df4d962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } }
I20260929 15:49:55.957396 20901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:55.962860 20908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9225cb3bd40b4b46b821097c2df4d962. Latest consensus state: current_term: 1 leader_uuid: "9225cb3bd40b4b46b821097c2df4d962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9225cb3bd40b4b46b821097c2df4d962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } } }
I20260929 15:49:55.963413 20908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:55.964035 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9225cb3bd40b4b46b821097c2df4d962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9225cb3bd40b4b46b821097c2df4d962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44083 } } }
I20260929 15:49:55.964423 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962 [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:55.979329 20919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:55.984985 20919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:55.995922 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:55.962322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37247 (local address 127.0.0.1:44083)
0929 15:49:55.962501 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:55.962510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:55.977130 (+ 14620us) server_negotiation.cc:407] Connection header received
0929 15:49:55.977725 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:55.977734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:55.977844 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:55.977966 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:55.980143 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:55.981021 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:55.983039 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:55.983767 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:55.983902 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:55.987390 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:55.987398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:55.987403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:49:55.987449 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:55.991579 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:55.992819 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:55.992829 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:55.992835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:55.993062 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:49:55.993684 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:49:55.993690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:49:55.993694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:49:55.994222 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:55.994529 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:55.995255 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 15:49:55.995534 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 15:49:55.996330 20919 catalog_manager.cc:1347] Generated new cluster ID: 377060a659dd49639ec1d4bc31334ff1
I20260929 15:49:55.996440 20919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:56.014199 20919 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:56.015609 20919 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:56.030586 20919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9225cb3bd40b4b46b821097c2df4d962: Generated new TSK 0
I20260929 15:49:56.031526 20919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:49:56.043943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20834
I20260929 15:49:56.045362 20896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:49:56.144646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20834
W20260929 15:49:56.164091 20819 connection.cc:535] client connection to 127.0.0.1:44083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44083): Connection reset by peer (error 104)
2026-09-29T15:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:49:56 dist-test-slave-7491 krb5kdc[20933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:49:56 dist-test-slave-7491 krb5kdc[20933](info): set up 2 sockets
Sep 29 15:49:56 dist-test-slave-7491 krb5kdc[20933](info): commencing operation
krb5kdc: starting...
W20260929 15:49:57.256585 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:49:57 dist-test-slave-7491 krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:49:57.523461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:57.871335 20946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:57.871639 20946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:57.871733 20946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:57.879467 20946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:57.879606 20946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:57.879655 20946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:57.879717 20946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:57.879762 20946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:57.890462 20946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:57.891974 20946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:57.893389 20946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:57.905913 20952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:57.906015 20951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:57.906513 20954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:57.906855 20946 server_base.cc:1034] running on GCE node
I20260929 15:49:57.907651 20946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:57.909196 20946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:57.910415 20946 hybrid_clock.cc:648] HybridClock initialized: now 1790696997910378 us; error 49 us; skew 500 ppm
I20260929 15:49:57.910808 20946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:57 dist-test-slave-7491 krb5kdc[20933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:57.914204 20946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:49:57.915925 20946 webserver.cc:460] Webserver started at http://127.0.0.1:36417/ using document root <none> and password file <none>
I20260929 15:49:57.916697 20946 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:57.916826 20946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:57.917171 20946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:57.920354 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "62f4aa3300824ef987a7cb78e9695fac"
format_stamp: "Formatted at 2026-09-29 15:49:57 on dist-test-slave-7491"
server_key: "adc3eee1d2a8d56817efe8e5d80daa7c"
server_key_iv: "61c950a75490e042daf2a56155ec58a3"
server_key_version: "encryptionkey@0"
I20260929 15:49:57.921192 20946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "62f4aa3300824ef987a7cb78e9695fac"
format_stamp: "Formatted at 2026-09-29 15:49:57 on dist-test-slave-7491"
server_key: "adc3eee1d2a8d56817efe8e5d80daa7c"
server_key_iv: "61c950a75490e042daf2a56155ec58a3"
server_key_version: "encryptionkey@0"
I20260929 15:49:57.926827 20946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:49:57.930734 20961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:57.932447 20946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:49:57.932659 20946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "62f4aa3300824ef987a7cb78e9695fac"
format_stamp: "Formatted at 2026-09-29 15:49:57 on dist-test-slave-7491"
server_key: "adc3eee1d2a8d56817efe8e5d80daa7c"
server_key_iv: "61c950a75490e042daf2a56155ec58a3"
server_key_version: "encryptionkey@0"
I20260929 15:49:57.932885 20946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:57.955179 20946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:57.960132 20946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:57.960685 20946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:57.981009 20946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44643
I20260929 15:49:57.981040 21012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44643 every 8 connection(s)
I20260929 15:49:57.982425 20946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:57.986263 21013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:57.990051 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 20946
I20260929 15:49:57.990272 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:57.990710 388 external_mini_cluster.cc:1428] Setting key 87e9c4cbf882ff423dc5c2cff2278056
I20260929 15:49:57.999382 21013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac: Bootstrap starting.
I20260929 15:49:58.003093 21013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:58.004437 21013 log.cc:824] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:58.004932 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:57.992665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53327 (local address 127.0.0.1:44643)
0929 15:49:57.993503 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:57.993515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:57.993559 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:49:57.995278 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:57.995301 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:57.995730 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:57.996187 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:57.996895 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:57.998362 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:57.999902 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:58.000730 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:58.000962 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:58.002379 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:58.002387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:58.002402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:58.002996 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:58.003093 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:58.003119 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:58.003307 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:58.003625 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 15:49:58.003803 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":392,"threads_started":1}
I20260929 15:49:58.007475 21013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac: No bootstrap required, opened a new log
I20260929 15:49:58.009222 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:49:58.017333 21013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aa3300824ef987a7cb78e9695fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260929 15:49:58.018288 21013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:58.018417 21013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f4aa3300824ef987a7cb78e9695fac, State: Initialized, Role: FOLLOWER
I20260929 15:49:58.019120 21013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aa3300824ef987a7cb78e9695fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260929 15:49:58.019405 21013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:58.019598 21013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:58.019795 21013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:58.021906 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:58.013994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51665 (local address 127.0.0.1:44643)
0929 15:49:58.014232 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:58.014238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:58.014562 (+ 324us) server_negotiation.cc:407] Connection header received
0929 15:49:58.014790 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:58.014797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:58.014869 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:58.014958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:58.015620 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:58.016468 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:58.018754 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:49:58.019347 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:49:58.019524 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:49:58.021061 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:58.021083 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:58.021088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:58.021113 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:58.021173 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:58.021197 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:58.021340 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:58.021555 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 15:49:58.021613 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 15:49:58.022648 21013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aa3300824ef987a7cb78e9695fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260929 15:49:58.023204 21013 leader_election.cc:302] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62f4aa3300824ef987a7cb78e9695fac; no voters:
I20260929 15:49:58.023840 21013 leader_election.cc:288] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:49:58.023981 21023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:49:58.024701 21023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [term 1 LEADER]: Becoming Leader. State: Replica: 62f4aa3300824ef987a7cb78e9695fac, State: Running, Role: LEADER
I20260929 15:49:58.025300 21023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aa3300824ef987a7cb78e9695fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260929 15:49:58.026047 21013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:49:58.027941 21025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62f4aa3300824ef987a7cb78e9695fac. Latest consensus state: current_term: 1 leader_uuid: "62f4aa3300824ef987a7cb78e9695fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aa3300824ef987a7cb78e9695fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260929 15:49:58.028478 21025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:58.028625 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62f4aa3300824ef987a7cb78e9695fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aa3300824ef987a7cb78e9695fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260929 15:49:58.028955 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac [sys.catalog]: This master's current role is: LEADER
I20260929 15:49:58.033463 21028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:49:58.041548 21028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:49:58.052304 21028 catalog_manager.cc:1347] Generated new cluster ID: a9c8b22d35ef4e669fa07010c5aee5d2
I20260929 15:49:58.052439 21028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:49:58.066028 21028 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:49:58.067574 21028 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:49:58.074209 21028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62f4aa3300824ef987a7cb78e9695fac: Generated new TSK 0
I20260929 15:49:58.075150 21028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:49:58.082180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20946
W20260929 15:49:58.084009 21008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53327
W20260929 15:49:58.084951 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 20946: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:49:58.085145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 20946
W20260929 15:49:58.103976 20931 connection.cc:535] client connection to 127.0.0.1:44643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44643): Connection reset by peer (error 104)
2026-09-29T15:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:49:58 dist-test-slave-7491 krb5kdc[21044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:49:58 dist-test-slave-7491 krb5kdc[21044](info): set up 2 sockets
Sep 29 15:49:58 dist-test-slave-7491 krb5kdc[21044](info): commencing operation
krb5kdc: starting...
W20260929 15:49:59.192705 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:49:59 dist-test-slave-7491 krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:49:59.463757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:49:59.851230 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:49:59.851684 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:49:59.851831 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:49:59.861605 21057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:49:59.861750 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:49:59.861801 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:49:59.861866 21057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:49:59.861919 21057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:49:59.873257 21057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:49:59.874778 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:49:59.876324 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:49:59.889087 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:59.889312 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:49:59.889361 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:49:59.890519 21057 server_base.cc:1034] running on GCE node
I20260929 15:49:59.891650 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:49:59.893632 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:49:59.895015 21057 hybrid_clock.cc:648] HybridClock initialized: now 1790696999894945 us; error 65 us; skew 500 ppm
I20260929 15:49:59.895761 21057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:49:59 dist-test-slave-7491 krb5kdc[21044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790696999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:49:59.900625 21057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:49:59.902518 21057 webserver.cc:460] Webserver started at http://127.0.0.1:37225/ using document root <none> and password file <none>
I20260929 15:49:59.903357 21057 fs_manager.cc:359] Metadata directory not provided
I20260929 15:49:59.903520 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:49:59.903863 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:49:59.907198 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "8b547032f3144457bcb61ebece9fb2e4"
format_stamp: "Formatted at 2026-09-29 15:49:59 on dist-test-slave-7491"
server_key: "f3415e0dd31c97893853cad52c1695b0"
server_key_iv: "e2b579f5e8881591edf6a5a39faa675a"
server_key_version: "encryptionkey@0"
I20260929 15:49:59.908099 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b547032f3144457bcb61ebece9fb2e4"
format_stamp: "Formatted at 2026-09-29 15:49:59 on dist-test-slave-7491"
server_key: "f3415e0dd31c97893853cad52c1695b0"
server_key_iv: "e2b579f5e8881591edf6a5a39faa675a"
server_key_version: "encryptionkey@0"
I20260929 15:49:59.914140 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 15:49:59.918432 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:49:59.920148 21057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:49:59.920445 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "8b547032f3144457bcb61ebece9fb2e4"
format_stamp: "Formatted at 2026-09-29 15:49:59 on dist-test-slave-7491"
server_key: "f3415e0dd31c97893853cad52c1695b0"
server_key_iv: "e2b579f5e8881591edf6a5a39faa675a"
server_key_version: "encryptionkey@0"
I20260929 15:49:59.920689 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:49:59.927273 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:49:59.934640 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:49:59.935041 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:49:59.956292 21057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37755
I20260929 15:49:59.956350 21123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37755 every 8 connection(s)
I20260929 15:49:59.957796 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:49:59.961803 21124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:49:59.962061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21057
I20260929 15:49:59.962261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:49:59.962718 388 external_mini_cluster.cc:1428] Setting key d96b7427f936bda31279e0ff063cbf9a
I20260929 15:49:59.971601 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:59.964488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54799 (local address 127.0.0.1:37755)
0929 15:49:59.965328 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:59.965340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:59.965388 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:49:59.967455 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:59.967487 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:59.967916 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:59.968384 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:59.969752 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:59.969762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:59.969776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:59.969804 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:59.969892 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:59.969931 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:59.970161 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:59.970338 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 15:49:59.970588 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":369,"threads_started":1}
I20260929 15:49:59.976330 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:49:59.977747 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4: Bootstrap starting.
I20260929 15:49:59.981541 21124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4: Neither blocks nor log segments found. Creating new log.
I20260929 15:49:59.982847 21124 log.cc:824] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:49:59.983644 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:49:59.980882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47175 (local address 127.0.0.1:37755)
0929 15:49:59.981076 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:49:59.981083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:49:59.981492 (+ 409us) server_negotiation.cc:407] Connection header received
0929 15:49:59.981709 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:49:59.981715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:49:59.981778 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:49:59.981891 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:49:59.982938 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:49:59.982944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:49:59.982955 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:49:59.982972 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:49:59.983037 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:49:59.983058 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:49:59.983174 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:49:59.983282 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 15:49:59.983348 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:49:59.986382 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4: No bootstrap required, opened a new log
I20260929 15:49:59.994536 21124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b547032f3144457bcb61ebece9fb2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260929 15:49:59.995038 21124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:49:59.995151 21124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b547032f3144457bcb61ebece9fb2e4, State: Initialized, Role: FOLLOWER
I20260929 15:49:59.995834 21124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b547032f3144457bcb61ebece9fb2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260929 15:49:59.996124 21124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:49:59.996330 21124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:49:59.996533 21124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:49:59.999207 21124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b547032f3144457bcb61ebece9fb2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260929 15:49:59.999869 21124 leader_election.cc:302] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b547032f3144457bcb61ebece9fb2e4; no voters:
I20260929 15:50:00.000721 21124 leader_election.cc:288] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:00.000893 21134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:00.001691 21134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 8b547032f3144457bcb61ebece9fb2e4, State: Running, Role: LEADER
I20260929 15:50:00.002249 21134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b547032f3144457bcb61ebece9fb2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260929 15:50:00.003144 21124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:00.005429 21136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b547032f3144457bcb61ebece9fb2e4. Latest consensus state: current_term: 1 leader_uuid: "8b547032f3144457bcb61ebece9fb2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b547032f3144457bcb61ebece9fb2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } } }
I20260929 15:50:00.005862 21136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:00.005600 21135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b547032f3144457bcb61ebece9fb2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b547032f3144457bcb61ebece9fb2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } } }
I20260929 15:50:00.005990 21135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:00.007938 21143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:00.016130 21143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:00.026988 21143 catalog_manager.cc:1347] Generated new cluster ID: 165617f741884a82884c2d0bae534ff1
I20260929 15:50:00.027153 21143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:00.078681 21143 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:00.079376 21143 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:00.089471 21143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b547032f3144457bcb61ebece9fb2e4: Generated new TSK 0
I20260929 15:50:00.090669 21143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:00.104598 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21057
W20260929 15:50:00.105816 21119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54799
W20260929 15:50:00.106598 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:00.106760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21057
2026-09-29T15:50:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:00 dist-test-slave-7491 krb5kdc[21155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:00 dist-test-slave-7491 krb5kdc[21155](info): set up 2 sockets
Sep 29 15:50:00 dist-test-slave-7491 krb5kdc[21155](info): commencing operation
krb5kdc: starting...
W20260929 15:50:01.248948 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:01 dist-test-slave-7491 krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:01.557101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:01.942848 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:01.943239 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:01.943428 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:01.951531 21168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:01.951681 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:01.951771 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:01.951874 21168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:01.951939 21168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:01.962929 21168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:01.964537 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:01.966058 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:01.980408 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:01.980343 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:01.980388 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:01.986618 21168 server_base.cc:1034] running on GCE node
I20260929 15:50:01.987628 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:01.989398 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:01.990728 21168 hybrid_clock.cc:648] HybridClock initialized: now 1790697001990681 us; error 47 us; skew 500 ppm
I20260929 15:50:01.991212 21168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:01 dist-test-slave-7491 krb5kdc[21155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:01.995666 21168 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:01.998049 21168 webserver.cc:460] Webserver started at http://127.0.0.1:41569/ using document root <none> and password file <none>
I20260929 15:50:01.998952 21168 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:01.999125 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:01.999497 21168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:02.003006 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "608c63f144014e7dbeb869f825309e9e"
format_stamp: "Formatted at 2026-09-29 15:50:01 on dist-test-slave-7491"
server_key: "e128f887ac7d672c3d2559f273c1999e"
server_key_iv: "2833613a99d941c30766a96ab9f6c6dd"
server_key_version: "encryptionkey@0"
I20260929 15:50:02.003933 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "608c63f144014e7dbeb869f825309e9e"
format_stamp: "Formatted at 2026-09-29 15:50:01 on dist-test-slave-7491"
server_key: "e128f887ac7d672c3d2559f273c1999e"
server_key_iv: "2833613a99d941c30766a96ab9f6c6dd"
server_key_version: "encryptionkey@0"
I20260929 15:50:02.010079 21168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 15:50:02.014856 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:02.016776 21168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:50:02.016971 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "608c63f144014e7dbeb869f825309e9e"
format_stamp: "Formatted at 2026-09-29 15:50:01 on dist-test-slave-7491"
server_key: "e128f887ac7d672c3d2559f273c1999e"
server_key_iv: "2833613a99d941c30766a96ab9f6c6dd"
server_key_version: "encryptionkey@0"
I20260929 15:50:02.017205 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:02.038038 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:02.043584 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:02.043957 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:02.067766 21168 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46639
I20260929 15:50:02.067790 21234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46639 every 8 connection(s)
I20260929 15:50:02.069423 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:02.074910 21235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:02.079443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21168
I20260929 15:50:02.079664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:02.080214 388 external_mini_cluster.cc:1428] Setting key cb02d2ad86574d06170f73d859ebb3b4
I20260929 15:50:02.088135 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e: Bootstrap starting.
I20260929 15:50:02.091643 21235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:02.093526 21235 log.cc:824] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:50:02 dist-test-slave-7491 krb5kdc[21155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:50:02.098819 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e: No bootstrap required, opened a new log
I20260929 15:50:02.109854 21235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608c63f144014e7dbeb869f825309e9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260929 15:50:02.110431 21235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:02.110594 21235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608c63f144014e7dbeb869f825309e9e, State: Initialized, Role: FOLLOWER
I20260929 15:50:02.111368 21235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608c63f144014e7dbeb869f825309e9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260929 15:50:02.111824 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:02.082466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42711 (local address 127.0.0.1:46639)
0929 15:50:02.083637 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:02.083768 (+ 131us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:02.083829 (+ 61us) server_negotiation.cc:407] Connection header received
0929 15:50:02.086654 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:02.086686 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:02.087335 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:02.087863 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:02.089545 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:02.090959 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:02.093034 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:02.093897 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:02.094120 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:02.100336 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:02.100349 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:02.100374 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:02.100435 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:02.105272 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:02.106604 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:02.106612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:02.106621 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:02.106870 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:02.107457 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:02.107461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:02.107463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:02.108273 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:02.108573 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:02.109369 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 15:50:02.110075 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":491,"threads_started":1}
I20260929 15:50:02.111822 21235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:02.112260 21235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:02.112471 21235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:02.115168 21235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608c63f144014e7dbeb869f825309e9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260929 15:50:02.115793 21235 leader_election.cc:302] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 608c63f144014e7dbeb869f825309e9e; no voters:
I20260929 15:50:02.116392 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:02.116511 21235 leader_election.cc:288] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:02.116667 21240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:02.117311 21240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [term 1 LEADER]: Becoming Leader. State: Replica: 608c63f144014e7dbeb869f825309e9e, State: Running, Role: LEADER
I20260929 15:50:02.117954 21240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608c63f144014e7dbeb869f825309e9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } }
I20260929 15:50:02.119844 21240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608c63f144014e7dbeb869f825309e9e. Latest consensus state: current_term: 1 leader_uuid: "608c63f144014e7dbeb869f825309e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608c63f144014e7dbeb869f825309e9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } } }
I20260929 15:50:02.120309 21240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:02.120733 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "608c63f144014e7dbeb869f825309e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608c63f144014e7dbeb869f825309e9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46639 } } }
I20260929 15:50:02.121526 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:02.132592 21235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:02.133114 21245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:02.139223 21245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:02.155874 21245 catalog_manager.cc:1347] Generated new cluster ID: 49da55f4316f41fbb3ffaf6ff8077774
I20260929 15:50:02.156133 21245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:02.162137 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:02.132507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34883 (local address 127.0.0.1:46639)
0929 15:50:02.132750 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:02.132760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:02.133267 (+ 507us) server_negotiation.cc:407] Connection header received
0929 15:50:02.133587 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:02.133593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:02.133688 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:02.133813 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:02.136826 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:02.138897 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:02.142295 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:02.143486 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:02.143678 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:02.149431 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:02.149446 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:02.149455 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:02.149533 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:02.155816 (+ 6283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:02.157259 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:02.157272 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:02.157278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:02.157510 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:02.158130 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:02.158138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:02.158144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:02.158753 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:02.159122 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:02.160815 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0929 15:50:02.161201 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":76}
I20260929 15:50:02.174539 21245 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:02.176638 21245 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:02.186673 21245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 608c63f144014e7dbeb869f825309e9e: Generated new TSK 0
I20260929 15:50:02.187696 21245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:02.198858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21168
I20260929 15:50:02.200726 21230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:02.298270 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21168
2026-09-29T15:50:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:02 dist-test-slave-7491 krb5kdc[21267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:02 dist-test-slave-7491 krb5kdc[21267](info): set up 2 sockets
Sep 29 15:50:02 dist-test-slave-7491 krb5kdc[21267](info): commencing operation
krb5kdc: starting...
W20260929 15:50:03.472740 388 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:03 dist-test-slave-7491 krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:03.767625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:04.138322 21280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:04.138715 21280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:04.138882 21280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:04.146701 21280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:04.146886 21280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:04.147009 21280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:04.147121 21280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:04.147202 21280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:04.157855 21280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:04.159328 21280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:04.160828 21280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:04.174886 21288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:04.175145 21285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:04.175135 21286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:04.176517 21280 server_base.cc:1034] running on GCE node
I20260929 15:50:04.177616 21280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:04.179438 21280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:04.180716 21280 hybrid_clock.cc:648] HybridClock initialized: now 1790697004180653 us; error 72 us; skew 500 ppm
I20260929 15:50:04.181169 21280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:04 dist-test-slave-7491 krb5kdc[21267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:04.184994 21280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:04.187466 21280 webserver.cc:460] Webserver started at http://127.0.0.1:33645/ using document root <none> and password file <none>
I20260929 15:50:04.188709 21280 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:04.188874 21280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:04.189301 21280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:04.193540 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "6aa47f374ee64e48a6dcda84ef735331"
format_stamp: "Formatted at 2026-09-29 15:50:04 on dist-test-slave-7491"
server_key: "3592be9db171ea25e04f3a2494357745"
server_key_iv: "c6dc0d12fceaba628f4028ced11ee7da"
server_key_version: "encryptionkey@0"
I20260929 15:50:04.194797 21280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "6aa47f374ee64e48a6dcda84ef735331"
format_stamp: "Formatted at 2026-09-29 15:50:04 on dist-test-slave-7491"
server_key: "3592be9db171ea25e04f3a2494357745"
server_key_iv: "c6dc0d12fceaba628f4028ced11ee7da"
server_key_version: "encryptionkey@0"
I20260929 15:50:04.202466 21280 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 15:50:04.207641 21295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:04.209887 21280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:50:04.210146 21280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "6aa47f374ee64e48a6dcda84ef735331"
format_stamp: "Formatted at 2026-09-29 15:50:04 on dist-test-slave-7491"
server_key: "3592be9db171ea25e04f3a2494357745"
server_key_iv: "c6dc0d12fceaba628f4028ced11ee7da"
server_key_version: "encryptionkey@0"
I20260929 15:50:04.210403 21280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:04.262929 21280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:04.267999 21280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:04.268396 21280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:04.292328 21280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37723
I20260929 15:50:04.292407 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37723 every 8 connection(s)
I20260929 15:50:04.294246 21280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:04.294625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21280
I20260929 15:50:04.294813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:04.295296 388 external_mini_cluster.cc:1428] Setting key 1fb894b79b5bc00fca65100ebe1f5d6f
I20260929 15:50:04.300781 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:04.312049 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:04.297995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46273 (local address 127.0.0.1:37723)
0929 15:50:04.298890 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:04.298903 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:04.298950 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:50:04.301116 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:04.301145 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:04.301632 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:04.302010 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:04.302722 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:04.304313 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:04.306389 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:04.307409 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:04.307611 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:04.309225 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:04.309235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:04.309254 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:04.309962 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:04.310059 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:04.310093 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:04.310213 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:04.310554 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 15:50:04.310913 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":412,"threads_started":1}
I20260929 15:50:04.315685 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331: Bootstrap starting.
I20260929 15:50:04.317631 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:04.321622 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:04.323524 21347 log.cc:824] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:04.327802 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331: No bootstrap required, opened a new log
I20260929 15:50:04.333978 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:04.325129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47039 (local address 127.0.0.1:37723)
0929 15:50:04.325356 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:04.325364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:04.325737 (+ 373us) server_negotiation.cc:407] Connection header received
0929 15:50:04.325953 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:04.325957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:04.326026 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:04.326126 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:04.326899 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:04.327984 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:04.330375 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:04.331081 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:04.331224 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:04.332947 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:04.332953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:04.332957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:04.332981 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:04.333037 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:04.333059 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:04.333157 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:04.333367 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 15:50:04.333470 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:50:04.339558 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa47f374ee64e48a6dcda84ef735331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260929 15:50:04.340008 21347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:04.340147 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa47f374ee64e48a6dcda84ef735331, State: Initialized, Role: FOLLOWER
I20260929 15:50:04.340840 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa47f374ee64e48a6dcda84ef735331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260929 15:50:04.341099 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:04.341275 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:04.341449 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:04.343863 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa47f374ee64e48a6dcda84ef735331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260929 15:50:04.344606 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aa47f374ee64e48a6dcda84ef735331; no voters:
I20260929 15:50:04.345315 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:04.345676 21357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:04.346655 21357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa47f374ee64e48a6dcda84ef735331, State: Running, Role: LEADER
I20260929 15:50:04.347259 21357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa47f374ee64e48a6dcda84ef735331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260929 15:50:04.348135 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:04.351532 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aa47f374ee64e48a6dcda84ef735331. Latest consensus state: current_term: 1 leader_uuid: "6aa47f374ee64e48a6dcda84ef735331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa47f374ee64e48a6dcda84ef735331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } } }
I20260929 15:50:04.352063 21358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aa47f374ee64e48a6dcda84ef735331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aa47f374ee64e48a6dcda84ef735331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } } }
I20260929 15:50:04.352356 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:04.352483 21358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:04.356588 21366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:04.361655 21366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:04.372501 21366 catalog_manager.cc:1347] Generated new cluster ID: 4bbd00c7ce9c471482af1a04a0ec860f
I20260929 15:50:04.372649 21366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:04.390944 21366 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:04.392452 21366 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:04.400852 21366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aa47f374ee64e48a6dcda84ef735331: Generated new TSK 0
I20260929 15:50:04.402148 21366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:04.420747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21280
W20260929 15:50:04.422444 21342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46273
W20260929 15:50:04.423388 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21280: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:04.423734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21280
2026-09-29T15:50:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:04 dist-test-slave-7491 krb5kdc[21378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:04 dist-test-slave-7491 krb5kdc[21378](info): set up 2 sockets
Sep 29 15:50:04 dist-test-slave-7491 krb5kdc[21378](info): commencing operation
krb5kdc: starting...
W20260929 15:50:05.556746 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:05 dist-test-slave-7491 krb5kdc[21378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:05.844666 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:06.207247 21391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:06.207672 21391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:06.207844 21391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:06.215788 21391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:06.215989 21391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:06.216104 21391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:06.216223 21391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:06.216298 21391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:06.228655 21391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:06.230216 21391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:06.231669 21391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:06.246637 21396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:06.246593 21397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:06.246672 21399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:06.248068 21391 server_base.cc:1034] running on GCE node
I20260929 15:50:06.249135 21391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:06.250759 21391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:06.252043 21391 hybrid_clock.cc:648] HybridClock initialized: now 1790697006252002 us; error 48 us; skew 500 ppm
I20260929 15:50:06.252610 21391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:06 dist-test-slave-7491 krb5kdc[21378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:06.264698 21391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:06.266615 21391 webserver.cc:460] Webserver started at http://127.0.0.1:43899/ using document root <none> and password file <none>
I20260929 15:50:06.267416 21391 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:06.267576 21391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:06.267964 21391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:06.271660 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "98f31072f4744a3d95625105f3a74ea4"
format_stamp: "Formatted at 2026-09-29 15:50:06 on dist-test-slave-7491"
server_key: "7c68307e07c978c295d8dfc503c57aa4"
server_key_iv: "ba504776286b60c5eea887c1134a0126"
server_key_version: "encryptionkey@0"
I20260929 15:50:06.272701 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "98f31072f4744a3d95625105f3a74ea4"
format_stamp: "Formatted at 2026-09-29 15:50:06 on dist-test-slave-7491"
server_key: "7c68307e07c978c295d8dfc503c57aa4"
server_key_iv: "ba504776286b60c5eea887c1134a0126"
server_key_version: "encryptionkey@0"
I20260929 15:50:06.278702 21391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:50:06.283219 21406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:06.285177 21391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 15:50:06.285420 21391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "98f31072f4744a3d95625105f3a74ea4"
format_stamp: "Formatted at 2026-09-29 15:50:06 on dist-test-slave-7491"
server_key: "7c68307e07c978c295d8dfc503c57aa4"
server_key_iv: "ba504776286b60c5eea887c1134a0126"
server_key_version: "encryptionkey@0"
I20260929 15:50:06.285668 21391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:06.293351 21391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:06.299131 21391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:06.299612 21391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:06.322115 21391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42927
I20260929 15:50:06.322165 21457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42927 every 8 connection(s)
I20260929 15:50:06.323721 21391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:06.325547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21391
I20260929 15:50:06.325729 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:06.326283 388 external_mini_cluster.cc:1428] Setting key 56421a542de352e8bff2f5ef29ef508e
I20260929 15:50:06.328912 21458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:06.336062 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:06.328292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58357 (local address 127.0.0.1:42927)
0929 15:50:06.329228 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:06.329241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:06.329289 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:50:06.331020 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:06.331042 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:06.331427 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:06.331869 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:06.333008 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:06.333014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:06.333028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:06.333043 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:06.333120 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:06.333141 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:06.333434 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:06.333592 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 15:50:06.334663 (+ 1071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":375,"threads_started":1}
I20260929 15:50:06.341068 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:06.343621 21458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4: Bootstrap starting.
I20260929 15:50:06.348068 21458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:06.348549 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:06.345726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54439 (local address 127.0.0.1:42927)
0929 15:50:06.345965 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:06.345973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:06.346363 (+ 390us) server_negotiation.cc:407] Connection header received
0929 15:50:06.346582 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:06.346588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:06.346648 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:06.346761 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:06.347793 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:06.347799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:06.347804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:06.347823 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:06.347876 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:06.347892 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:06.348032 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:06.348155 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 15:50:06.348251 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:50:06.349620 21458 log.cc:824] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:06.353317 21458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4: No bootstrap required, opened a new log
I20260929 15:50:06.361836 21458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f31072f4744a3d95625105f3a74ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260929 15:50:06.362342 21458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:06.362463 21458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f31072f4744a3d95625105f3a74ea4, State: Initialized, Role: FOLLOWER
I20260929 15:50:06.363130 21458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f31072f4744a3d95625105f3a74ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260929 15:50:06.363412 21458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:06.363575 21458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:06.363758 21458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:06.366281 21458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f31072f4744a3d95625105f3a74ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260929 15:50:06.366868 21458 leader_election.cc:302] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f31072f4744a3d95625105f3a74ea4; no voters:
I20260929 15:50:06.367740 21458 leader_election.cc:288] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:06.367779 21468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:06.368598 21468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [term 1 LEADER]: Becoming Leader. State: Replica: 98f31072f4744a3d95625105f3a74ea4, State: Running, Role: LEADER
I20260929 15:50:06.369186 21468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f31072f4744a3d95625105f3a74ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } }
I20260929 15:50:06.370038 21458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:06.372242 21469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f31072f4744a3d95625105f3a74ea4. Latest consensus state: current_term: 1 leader_uuid: "98f31072f4744a3d95625105f3a74ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f31072f4744a3d95625105f3a74ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } } }
I20260929 15:50:06.372272 21470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f31072f4744a3d95625105f3a74ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f31072f4744a3d95625105f3a74ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42927 } } }
I20260929 15:50:06.372757 21469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:06.372783 21470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:06.375036 21474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:06.380921 21474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:06.392107 21474 catalog_manager.cc:1347] Generated new cluster ID: a088eaa78916462389a59986482693b3
I20260929 15:50:06.392266 21474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:06.419739 21474 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:06.420406 21474 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:06.436985 21474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f31072f4744a3d95625105f3a74ea4: Generated new TSK 0
I20260929 15:50:06.438392 21474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:06.447203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21391
W20260929 15:50:06.448455 21453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58357
W20260929 15:50:06.449234 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:06.449407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21391
2026-09-29T15:50:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:06 dist-test-slave-7491 krb5kdc[21489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:06 dist-test-slave-7491 krb5kdc[21489](info): set up 2 sockets
Sep 29 15:50:06 dist-test-slave-7491 krb5kdc[21489](info): commencing operation
krb5kdc: starting...
W20260929 15:50:07.612962 388 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:07 dist-test-slave-7491 krb5kdc[21489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:07.908586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:08.279913 21502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:08.280359 21502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:08.280551 21502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:08.288458 21502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:08.288594 21502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:08.288692 21502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:08.288780 21502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:08.288892 21502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:08.299809 21502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:08.301322 21502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:08.302732 21502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:08.315884 21508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:08.316347 21507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:08.315884 21510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:08.317358 21502 server_base.cc:1034] running on GCE node
I20260929 15:50:08.318252 21502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:08.319888 21502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:08.321156 21502 hybrid_clock.cc:648] HybridClock initialized: now 1790697008321075 us; error 78 us; skew 500 ppm
I20260929 15:50:08.321612 21502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:08 dist-test-slave-7491 krb5kdc[21489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:08.325266 21502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:08.327109 21502 webserver.cc:460] Webserver started at http://127.0.0.1:36791/ using document root <none> and password file <none>
I20260929 15:50:08.327931 21502 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:08.328104 21502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:08.328471 21502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:08.332620 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "2a5934552c794b12a064506702e7c115"
format_stamp: "Formatted at 2026-09-29 15:50:08 on dist-test-slave-7491"
server_key: "aec18824a3377e77110f6097b9196fb9"
server_key_iv: "8d786aad93cd433114bc5fc51e84c1fc"
server_key_version: "encryptionkey@0"
I20260929 15:50:08.333581 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a5934552c794b12a064506702e7c115"
format_stamp: "Formatted at 2026-09-29 15:50:08 on dist-test-slave-7491"
server_key: "aec18824a3377e77110f6097b9196fb9"
server_key_iv: "8d786aad93cd433114bc5fc51e84c1fc"
server_key_version: "encryptionkey@0"
I20260929 15:50:08.340137 21502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 15:50:08.344152 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:08.345955 21502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:50:08.346194 21502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "2a5934552c794b12a064506702e7c115"
format_stamp: "Formatted at 2026-09-29 15:50:08 on dist-test-slave-7491"
server_key: "aec18824a3377e77110f6097b9196fb9"
server_key_iv: "8d786aad93cd433114bc5fc51e84c1fc"
server_key_version: "encryptionkey@0"
I20260929 15:50:08.346453 21502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:08.380062 21502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:08.385604 21502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:08.385960 21502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:08.410408 21502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46439
I20260929 15:50:08.410678 21568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46439 every 8 connection(s)
I20260929 15:50:08.412340 21502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:08.416990 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:08.419085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21502
I20260929 15:50:08.419305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:08.419749 388 external_mini_cluster.cc:1428] Setting key 84eba20e891d545d3b254abd93334593
I20260929 15:50:08.432806 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115: Bootstrap starting.
Sep 29 15:50:08 dist-test-slave-7491 krb5kdc[21489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:50:08.436993 21569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:08.438220 21569 log.cc:824] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:08.441855 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115: No bootstrap required, opened a new log
I20260929 15:50:08.446512 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:08.421887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:34345 (local address 127.0.0.1:46439)
0929 15:50:08.422813 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:08.422828 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:08.422888 (+ 60us) server_negotiation.cc:407] Connection header received
0929 15:50:08.424754 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:08.424782 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:08.425311 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:08.425765 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:08.427357 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:08.428893 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:08.430319 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:08.431139 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:08.431369 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:08.437194 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:08.437210 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:08.437248 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:08.437412 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:08.441520 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:08.442681 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:08.442689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:08.442696 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:08.442893 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:08.443432 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:08.443437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:08.443439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:08.444044 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:08.444463 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:08.444995 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 15:50:08.445364 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":420,"threads_started":1}
I20260929 15:50:08.451784 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:08.452646 21569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5934552c794b12a064506702e7c115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46439 } }
I20260929 15:50:08.453100 21569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:08.453212 21569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5934552c794b12a064506702e7c115, State: Initialized, Role: FOLLOWER
I20260929 15:50:08.453863 21569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5934552c794b12a064506702e7c115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46439 } }
I20260929 15:50:08.454129 21569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:08.454281 21569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:08.454476 21569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:08.457574 21569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5934552c794b12a064506702e7c115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46439 } }
I20260929 15:50:08.458341 21569 leader_election.cc:302] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5934552c794b12a064506702e7c115; no voters:
I20260929 15:50:08.459223 21569 leader_election.cc:288] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:08.459409 21579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:08.460307 21579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5934552c794b12a064506702e7c115, State: Running, Role: LEADER
I20260929 15:50:08.461031 21579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5934552c794b12a064506702e7c115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46439 } }
I20260929 15:50:08.463429 21569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:08.464104 21580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a5934552c794b12a064506702e7c115. Latest consensus state: current_term: 1 leader_uuid: "2a5934552c794b12a064506702e7c115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5934552c794b12a064506702e7c115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46439 } } }
I20260929 15:50:08.464704 21580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:08.467875 21581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a5934552c794b12a064506702e7c115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5934552c794b12a064506702e7c115" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46439 } } }
I20260929 15:50:08.468355 21581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:08.470432 21583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:08.478494 21583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:08.481549 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:08.457180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:56955 (local address 127.0.0.1:46439)
0929 15:50:08.457528 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:08.457537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:08.458274 (+ 737us) server_negotiation.cc:407] Connection header received
0929 15:50:08.458499 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:08.458504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:08.458701 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:08.458875 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:08.461499 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:08.462767 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:08.464961 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:08.465899 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:08.466082 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:08.470126 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:08.470136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:08.470141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:08.470189 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:08.475754 (+ 5565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:08.477159 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:08.477171 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:08.477176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:08.477355 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:08.478048 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:08.478053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:08.478057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:08.479347 (+ 1290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:08.480695 (+ 1348us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:08.480903 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 15:50:08.481127 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260929 15:50:08.488652 21583 catalog_manager.cc:1347] Generated new cluster ID: 6797c306526f43d6a1e50dbee0f7dd10
I20260929 15:50:08.488831 21583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:08.507113 21583 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:08.508734 21583 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:08.518361 21583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a5934552c794b12a064506702e7c115: Generated new TSK 0
I20260929 15:50:08.519330 21583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:08.529023 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21502
I20260929 15:50:08.530297 21564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:08.626791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21502
W20260929 15:50:08.645076 21487 connection.cc:535] client connection to 127.0.0.1:46439 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46439): Connection reset by peer (error 104)
2026-09-29T15:50:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:08 dist-test-slave-7491 krb5kdc[21601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:08 dist-test-slave-7491 krb5kdc[21601](info): set up 2 sockets
Sep 29 15:50:08 dist-test-slave-7491 krb5kdc[21601](info): commencing operation
krb5kdc: starting...
W20260929 15:50:09.768707 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:09 dist-test-slave-7491 krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:10.067410 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:10.434489 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:10.434845 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:10.435017 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:10.443024 21614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:10.443183 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:10.443297 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:10.443552 21614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:10.443780 21614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:10.456429 21614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:10.458570 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:10.460158 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:10.473297 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:10.473438 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:10.473529 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:10.474403 21614 server_base.cc:1034] running on GCE node
I20260929 15:50:10.475342 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:10.477057 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:10.478314 21614 hybrid_clock.cc:648] HybridClock initialized: now 1790697010478268 us; error 36 us; skew 500 ppm
I20260929 15:50:10.478675 21614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:10 dist-test-slave-7491 krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:10.482155 21614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:10.483882 21614 webserver.cc:460] Webserver started at http://127.0.0.1:35709/ using document root <none> and password file <none>
I20260929 15:50:10.484683 21614 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:10.484794 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:10.485064 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:10.488291 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "77000db026e747de85598d3462f0980f"
format_stamp: "Formatted at 2026-09-29 15:50:10 on dist-test-slave-7491"
server_key: "29c6bbfa2b7eb1aa1d41532aae38a114"
server_key_iv: "ae304b391405868234484d1cd2f1ef6f"
server_key_version: "encryptionkey@0"
I20260929 15:50:10.489284 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "77000db026e747de85598d3462f0980f"
format_stamp: "Formatted at 2026-09-29 15:50:10 on dist-test-slave-7491"
server_key: "29c6bbfa2b7eb1aa1d41532aae38a114"
server_key_iv: "ae304b391405868234484d1cd2f1ef6f"
server_key_version: "encryptionkey@0"
I20260929 15:50:10.495219 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:50:10.499526 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:10.501216 21614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:50:10.501433 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "77000db026e747de85598d3462f0980f"
format_stamp: "Formatted at 2026-09-29 15:50:10 on dist-test-slave-7491"
server_key: "29c6bbfa2b7eb1aa1d41532aae38a114"
server_key_iv: "ae304b391405868234484d1cd2f1ef6f"
server_key_version: "encryptionkey@0"
I20260929 15:50:10.501684 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:10.520766 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:10.526342 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:10.526714 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:10.548542 21614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39353
I20260929 15:50:10.548564 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39353 every 8 connection(s)
I20260929 15:50:10.550052 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:10.555302 21681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:10.558790 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21614
I20260929 15:50:10.559008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:10.559443 388 external_mini_cluster.cc:1428] Setting key 03ec91d001549b80376b790084128b3e
I20260929 15:50:10.569242 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f: Bootstrap starting.
I20260929 15:50:10.573992 21681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:10.575330 21681 log.cc:824] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:10.576146 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:10.561525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:58027 (local address 127.0.0.1:39353)
0929 15:50:10.562349 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:10.562362 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:10.562406 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:50:10.564451 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:10.564479 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:10.564989 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:10.565381 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:10.566302 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:10.567985 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:10.570135 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:10.571010 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:10.571211 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:10.573234 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:10.573242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:10.573260 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:10.573864 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:10.573959 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:10.573996 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:10.574230 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:10.574627 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 15:50:10.574839 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":363,"threads_started":1}
I20260929 15:50:10.579695 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f: No bootstrap required, opened a new log
I20260929 15:50:10.581434 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:10.588634 21681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77000db026e747de85598d3462f0980f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260929 15:50:10.589293 21681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:10.589474 21681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77000db026e747de85598d3462f0980f, State: Initialized, Role: FOLLOWER
I20260929 15:50:10.590919 21681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77000db026e747de85598d3462f0980f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260929 15:50:10.591295 21681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:10.591452 21681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:10.591646 21681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:10.594985 21681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77000db026e747de85598d3462f0980f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260929 15:50:10.596293 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:10.587327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:50159 (local address 127.0.0.1:39353)
0929 15:50:10.587665 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:10.587672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:10.588129 (+ 457us) server_negotiation.cc:407] Connection header received
0929 15:50:10.588465 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:10.588470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:10.588544 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:10.588649 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:10.589378 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:10.590566 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:10.592826 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:10.593333 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:10.593472 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:10.594847 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:10.594853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:10.594856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:10.594878 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:10.594935 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:10.594954 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:10.595055 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:10.595916 (+ 861us) server_negotiation.cc:299] Negotiation successful
0929 15:50:10.595971 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 15:50:10.596266 21681 leader_election.cc:302] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77000db026e747de85598d3462f0980f; no voters:
I20260929 15:50:10.597184 21681 leader_election.cc:288] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:10.597229 21691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:10.599872 21681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:10.600523 21691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [term 1 LEADER]: Becoming Leader. State: Replica: 77000db026e747de85598d3462f0980f, State: Running, Role: LEADER
I20260929 15:50:10.601143 21691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77000db026e747de85598d3462f0980f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260929 15:50:10.604266 21693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77000db026e747de85598d3462f0980f. Latest consensus state: current_term: 1 leader_uuid: "77000db026e747de85598d3462f0980f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77000db026e747de85598d3462f0980f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } } }
I20260929 15:50:10.604692 21693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:10.604833 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77000db026e747de85598d3462f0980f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77000db026e747de85598d3462f0980f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } } }
I20260929 15:50:10.605131 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:10.608603 21701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:10.614395 21701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:10.626176 21701 catalog_manager.cc:1347] Generated new cluster ID: 37c1b59af4354fcbaecf7870757fe937
I20260929 15:50:10.626330 21701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:10.663452 21701 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:10.665578 21701 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:10.679322 21701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77000db026e747de85598d3462f0980f: Generated new TSK 0
I20260929 15:50:10.680415 21701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:10.700855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21614
W20260929 15:50:10.702312 21676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:58027
W20260929 15:50:10.703410 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:10.703614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21614
W20260929 15:50:10.722214 21599 connection.cc:535] client connection to 127.0.0.1:39353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39353): Connection reset by peer (error 104)
2026-09-29T15:50:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:10 dist-test-slave-7491 krb5kdc[21712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:10 dist-test-slave-7491 krb5kdc[21712](info): set up 2 sockets
Sep 29 15:50:10 dist-test-slave-7491 krb5kdc[21712](info): commencing operation
krb5kdc: starting...
W20260929 15:50:11.844794 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:12 dist-test-slave-7491 krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:12.144471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:12.497735 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:12.498066 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:12.498226 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:12.505805 21725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:12.505941 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:12.506028 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:12.506132 21725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:12.506225 21725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:12.516721 21725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:12.518177 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:12.519666 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:12.533452 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:12.533478 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:12.533525 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:12.534294 21725 server_base.cc:1034] running on GCE node
I20260929 15:50:12.535279 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:12.537133 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:12.538453 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790697012538367 us; error 73 us; skew 500 ppm
I20260929 15:50:12.538894 21725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:12 dist-test-slave-7491 krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:12.549261 21725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:12.551317 21725 webserver.cc:460] Webserver started at http://127.0.0.1:33297/ using document root <none> and password file <none>
I20260929 15:50:12.552148 21725 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:12.552388 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:12.552738 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:12.556061 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "41ce1f28ea2547668fac23d8fb7b0272"
format_stamp: "Formatted at 2026-09-29 15:50:12 on dist-test-slave-7491"
server_key: "612683dd85db9e4372a10bd5793be252"
server_key_iv: "5fadd5cf7e963e523744efd16a59e958"
server_key_version: "encryptionkey@0"
I20260929 15:50:12.557034 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "41ce1f28ea2547668fac23d8fb7b0272"
format_stamp: "Formatted at 2026-09-29 15:50:12 on dist-test-slave-7491"
server_key: "612683dd85db9e4372a10bd5793be252"
server_key_iv: "5fadd5cf7e963e523744efd16a59e958"
server_key_version: "encryptionkey@0"
I20260929 15:50:12.562988 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:50:12.567145 21740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:12.568953 21725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 15:50:12.569185 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "41ce1f28ea2547668fac23d8fb7b0272"
format_stamp: "Formatted at 2026-09-29 15:50:12 on dist-test-slave-7491"
server_key: "612683dd85db9e4372a10bd5793be252"
server_key_iv: "5fadd5cf7e963e523744efd16a59e958"
server_key_version: "encryptionkey@0"
I20260929 15:50:12.569600 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:12.577339 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:12.582347 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:12.582706 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:12.605525 21725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38307
I20260929 15:50:12.605543 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38307 every 8 connection(s)
I20260929 15:50:12.607116 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:12.611258 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:12.613655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21725
I20260929 15:50:12.614022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:12.614472 388 external_mini_cluster.cc:1428] Setting key 4b0ca9f7aff1b469588b21ff5311c878
I20260929 15:50:12.624491 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:12.616485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:58123 (local address 127.0.0.1:38307)
0929 15:50:12.617908 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:12.617926 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:12.618009 (+ 83us) server_negotiation.cc:407] Connection header received
0929 15:50:12.620533 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:12.620569 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:12.621040 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:12.621509 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:12.622696 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:12.622703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:12.622718 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:12.622734 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:12.622801 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:12.622829 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:12.623069 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:12.623212 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 15:50:12.623422 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":700,"threads_started":1}
I20260929 15:50:12.628306 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272: Bootstrap starting.
I20260929 15:50:12.629364 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:12.631871 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:12.633209 21792 log.cc:824] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:12.636943 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272: No bootstrap required, opened a new log
I20260929 15:50:12.638533 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:12.635073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:45013 (local address 127.0.0.1:38307)
0929 15:50:12.635317 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:12.635324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:12.635742 (+ 418us) server_negotiation.cc:407] Connection header received
0929 15:50:12.636021 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:12.636026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:12.636079 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:12.636271 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:12.637645 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:12.637655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:12.637661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:12.637693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:12.637748 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:12.637770 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:12.638009 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:12.638115 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 15:50:12.638175 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:50:12.646306 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ce1f28ea2547668fac23d8fb7b0272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 15:50:12.646853 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:12.646971 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ce1f28ea2547668fac23d8fb7b0272, State: Initialized, Role: FOLLOWER
I20260929 15:50:12.647709 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ce1f28ea2547668fac23d8fb7b0272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 15:50:12.647967 21792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:12.648144 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:12.648391 21792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:12.651430 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ce1f28ea2547668fac23d8fb7b0272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 15:50:12.652107 21792 leader_election.cc:302] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41ce1f28ea2547668fac23d8fb7b0272; no voters:
I20260929 15:50:12.652923 21792 leader_election.cc:288] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:12.653254 21802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:12.654206 21802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [term 1 LEADER]: Becoming Leader. State: Replica: 41ce1f28ea2547668fac23d8fb7b0272, State: Running, Role: LEADER
I20260929 15:50:12.654839 21802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ce1f28ea2547668fac23d8fb7b0272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 15:50:12.655519 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:12.658736 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41ce1f28ea2547668fac23d8fb7b0272. Latest consensus state: current_term: 1 leader_uuid: "41ce1f28ea2547668fac23d8fb7b0272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ce1f28ea2547668fac23d8fb7b0272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } } }
I20260929 15:50:12.659420 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:12.660235 21803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41ce1f28ea2547668fac23d8fb7b0272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ce1f28ea2547668fac23d8fb7b0272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } } }
I20260929 15:50:12.660660 21803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:12.660673 21809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:12.667100 21809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:12.679921 21809 catalog_manager.cc:1347] Generated new cluster ID: 55ae8140f3c645f19a8a18bafc88ae93
I20260929 15:50:12.680086 21809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:12.707245 21809 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:12.707868 21809 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:12.718626 21809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41ce1f28ea2547668fac23d8fb7b0272: Generated new TSK 0
I20260929 15:50:12.719983 21809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:12.725528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21725
W20260929 15:50:12.727793 21787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:58123
W20260929 15:50:12.728680 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:12.728864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21725
2026-09-29T15:50:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:12 dist-test-slave-7491 krb5kdc[21823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:12 dist-test-slave-7491 krb5kdc[21823](info): set up 2 sockets
Sep 29 15:50:12 dist-test-slave-7491 krb5kdc[21823](info): commencing operation
krb5kdc: starting...
W20260929 15:50:13.881361 388 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:14 dist-test-slave-7491 krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:14.171090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:14.531965 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:14.532374 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:14.532538 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:14.540897 21836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:14.541041 21836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:14.541314 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:14.541489 21836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:14.541715 21836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:14.554076 21836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:14.555583 21836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:14.557070 21836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:14.570087 21841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:14.570111 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:14.570065 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:14.571170 21836 server_base.cc:1034] running on GCE node
I20260929 15:50:14.572053 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:14.573715 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:14.575037 21836 hybrid_clock.cc:648] HybridClock initialized: now 1790697014574969 us; error 57 us; skew 500 ppm
I20260929 15:50:14.575470 21836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:14 dist-test-slave-7491 krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:14.579710 21836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:14.581723 21836 webserver.cc:460] Webserver started at http://127.0.0.1:45253/ using document root <none> and password file <none>
I20260929 15:50:14.582530 21836 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:14.582652 21836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:14.583004 21836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:14.586450 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "f97f8d2b73c444b0b59888b2e5647847"
format_stamp: "Formatted at 2026-09-29 15:50:14 on dist-test-slave-7491"
server_key: "e5afccc420286a0c2b4a020a3038db9d"
server_key_iv: "a4baa3705cc017a20b44ee211de85e26"
server_key_version: "encryptionkey@0"
I20260929 15:50:14.587354 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "f97f8d2b73c444b0b59888b2e5647847"
format_stamp: "Formatted at 2026-09-29 15:50:14 on dist-test-slave-7491"
server_key: "e5afccc420286a0c2b4a020a3038db9d"
server_key_iv: "a4baa3705cc017a20b44ee211de85e26"
server_key_version: "encryptionkey@0"
I20260929 15:50:14.594012 21836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 15:50:14.598451 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:14.600062 21836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:50:14.600333 21836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "f97f8d2b73c444b0b59888b2e5647847"
format_stamp: "Formatted at 2026-09-29 15:50:14 on dist-test-slave-7491"
server_key: "e5afccc420286a0c2b4a020a3038db9d"
server_key_iv: "a4baa3705cc017a20b44ee211de85e26"
server_key_version: "encryptionkey@0"
I20260929 15:50:14.600577 21836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:14.619374 21836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:14.624296 21836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:14.624625 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:14.646740 21836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45911
I20260929 15:50:14.646790 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45911 every 8 connection(s)
I20260929 15:50:14.648340 21836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:14.651938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21836
I20260929 15:50:14.652146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:14.652618 388 external_mini_cluster.cc:1428] Setting key cf85e6ee0a024026016028201a12f1b7
I20260929 15:50:14.653646 21903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:50:14 dist-test-slave-7491 krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 15:50:14.669603 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847: Bootstrap starting.
I20260929 15:50:14.674836 21903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:14.676405 21903 log.cc:824] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:14.681391 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847: No bootstrap required, opened a new log
I20260929 15:50:14.683012 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:14.654865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:50929 (local address 127.0.0.1:45911)
0929 15:50:14.655823 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:14.655839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:14.655893 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:50:14.658038 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:14.658069 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:14.658706 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:14.659204 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:14.660691 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:14.662810 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:14.664746 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:14.665862 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:14.666086 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:14.670989 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:14.671006 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:14.671034 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:14.671199 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:14.676159 (+ 4960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:14.678135 (+ 1976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:14.678147 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:14.678157 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:14.678439 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:14.679295 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:14.679302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:14.679306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:14.680128 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:14.680548 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:14.681104 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 15:50:14.681512 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":357,"threads_started":1}
I20260929 15:50:14.688009 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:14.692131 21903 raft_consensus.cc:348] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97f8d2b73c444b0b59888b2e5647847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260929 15:50:14.693377 21903 raft_consensus.cc:374] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:14.693558 21903 raft_consensus.cc:729] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97f8d2b73c444b0b59888b2e5647847, State: Initialized, Role: FOLLOWER
I20260929 15:50:14.694334 21903 consensus_queue.cc:260] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97f8d2b73c444b0b59888b2e5647847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260929 15:50:14.694645 21903 raft_consensus.cc:388] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:14.694972 21903 raft_consensus.cc:482] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:14.695176 21903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:14.697764 21903 raft_consensus.cc:504] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97f8d2b73c444b0b59888b2e5647847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260929 15:50:14.698829 21903 leader_election.cc:302] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f97f8d2b73c444b0b59888b2e5647847; no voters:
I20260929 15:50:14.699805 21903 leader_election.cc:288] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:14.699944 21913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:14.702617 21913 raft_consensus.cc:686] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [term 1 LEADER]: Becoming Leader. State: Replica: f97f8d2b73c444b0b59888b2e5647847, State: Running, Role: LEADER
I20260929 15:50:14.703447 21903 sys_catalog.cc:564] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:14.704897 21913 consensus_queue.cc:237] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97f8d2b73c444b0b59888b2e5647847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260929 15:50:14.708479 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f97f8d2b73c444b0b59888b2e5647847. Latest consensus state: current_term: 1 leader_uuid: "f97f8d2b73c444b0b59888b2e5647847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97f8d2b73c444b0b59888b2e5647847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260929 15:50:14.708890 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:14.709885 21919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:14.709662 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f97f8d2b73c444b0b59888b2e5647847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97f8d2b73c444b0b59888b2e5647847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260929 15:50:14.712100 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:14.714704 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:14.693257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:38247 (local address 127.0.0.1:45911)
0929 15:50:14.693438 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:14.693442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:14.694226 (+ 784us) server_negotiation.cc:407] Connection header received
0929 15:50:14.694465 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:14.694474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:14.694564 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:14.694776 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:14.696447 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:14.697275 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:14.698685 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:14.699218 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:14.699362 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:14.702448 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:14.702458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:14.702466 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:14.702544 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:14.707880 (+ 5336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:14.709278 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:14.709288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:14.709294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:14.709620 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:14.710635 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:14.710644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:14.710650 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:14.711305 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:14.713624 (+ 2319us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:14.713857 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 15:50:14.714165 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:50:14.724602 21919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:14.736617 21919 catalog_manager.cc:1347] Generated new cluster ID: 2898b2f8bbac43188ab170dd8e7ba2c0
I20260929 15:50:14.736769 21919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:14.758085 21919 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:14.759516 21919 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:14.778108 21919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f97f8d2b73c444b0b59888b2e5647847: Generated new TSK 0
I20260929 15:50:14.779080 21919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:14.786345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21836
I20260929 15:50:14.787792 21898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:14.883697 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21836
W20260929 15:50:14.903728 21821 connection.cc:535] client connection to 127.0.0.1:45911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45911): Connection reset by peer (error 104)
2026-09-29T15:50:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:14 dist-test-slave-7491 krb5kdc[21935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:14 dist-test-slave-7491 krb5kdc[21935](info): set up 2 sockets
Sep 29 15:50:14 dist-test-slave-7491 krb5kdc[21935](info): commencing operation
krb5kdc: starting...
W20260929 15:50:16.022408 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:16 dist-test-slave-7491 krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:16.307428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:16.685447 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:16.685776 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:16.685945 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:16.693625 21948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:16.693764 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:16.693813 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:16.693931 21948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:16.694015 21948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:16.704852 21948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:16.706583 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:16.708041 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:16.721045 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:16.721498 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:16.721614 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:16.722808 21948 server_base.cc:1034] running on GCE node
I20260929 15:50:16.724042 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:16.726107 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:16.727468 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790697016727372 us; error 81 us; skew 500 ppm
I20260929 15:50:16.727955 21948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:16 dist-test-slave-7491 krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:16.731825 21948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:16.733862 21948 webserver.cc:460] Webserver started at http://127.0.0.1:36721/ using document root <none> and password file <none>
I20260929 15:50:16.734659 21948 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:16.734822 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:16.735150 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:16.738550 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "6d134526421d4e5db983778ca31e6e18"
format_stamp: "Formatted at 2026-09-29 15:50:16 on dist-test-slave-7491"
server_key: "81d0ded9cf2866de3df653c0130618de"
server_key_iv: "62f7c93467abcf52ea642570bb08ae63"
server_key_version: "encryptionkey@0"
I20260929 15:50:16.739435 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d134526421d4e5db983778ca31e6e18"
format_stamp: "Formatted at 2026-09-29 15:50:16 on dist-test-slave-7491"
server_key: "81d0ded9cf2866de3df653c0130618de"
server_key_iv: "62f7c93467abcf52ea642570bb08ae63"
server_key_version: "encryptionkey@0"
I20260929 15:50:16.745352 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:50:16.749518 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:16.751152 21948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 15:50:16.751402 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "6d134526421d4e5db983778ca31e6e18"
format_stamp: "Formatted at 2026-09-29 15:50:16 on dist-test-slave-7491"
server_key: "81d0ded9cf2866de3df653c0130618de"
server_key_iv: "62f7c93467abcf52ea642570bb08ae63"
server_key_version: "encryptionkey@0"
I20260929 15:50:16.751641 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:16.786535 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:16.792623 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:16.792989 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:16.815415 21948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33469
I20260929 15:50:16.815445 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33469 every 8 connection(s)
I20260929 15:50:16.817027 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:16.817571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 21948
I20260929 15:50:16.817767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:16.818544 388 external_mini_cluster.cc:1428] Setting key abfaf4f3e5024cf417dc79ea392c32f4
I20260929 15:50:16.821368 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:16.837625 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18: Bootstrap starting.
I20260929 15:50:16.840902 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:16.825324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:45753 (local address 127.0.0.1:33469)
0929 15:50:16.826452 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:16.826470 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:16.826532 (+ 62us) server_negotiation.cc:407] Connection header received
0929 15:50:16.828937 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:16.828971 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:16.829604 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:16.830098 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:16.830628 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:16.832653 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:16.834123 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:16.835367 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:16.835572 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:16.837587 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:16.837605 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:16.837629 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:16.838439 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:16.838557 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:16.838595 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:16.838809 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:16.839167 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 15:50:16.839386 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":534,"threads_started":1}
I20260929 15:50:16.841985 22015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:16.843461 22015 log.cc:824] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:16.846140 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:16.847432 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18: No bootstrap required, opened a new log
I20260929 15:50:16.856493 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d134526421d4e5db983778ca31e6e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33469 } }
I20260929 15:50:16.857051 22015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:16.857213 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d134526421d4e5db983778ca31e6e18, State: Initialized, Role: FOLLOWER
I20260929 15:50:16.858008 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d134526421d4e5db983778ca31e6e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33469 } }
I20260929 15:50:16.858322 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:16.850590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:51797 (local address 127.0.0.1:33469)
0929 15:50:16.850764 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:16.850768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:16.851235 (+ 467us) server_negotiation.cc:407] Connection header received
0929 15:50:16.851485 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:16.851489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:16.851564 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:16.851692 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:16.852240 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:16.853070 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:16.854487 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:16.855273 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:16.855522 (+ 249us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:16.857322 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:16.857330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:16.857335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:16.857360 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:16.857443 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:16.857469 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:16.857607 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:16.857893 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 15:50:16.857959 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:50:16.858363 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:16.858521 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:16.858711 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:16.861485 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d134526421d4e5db983778ca31e6e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33469 } }
I20260929 15:50:16.862057 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d134526421d4e5db983778ca31e6e18; no voters:
I20260929 15:50:16.862700 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:16.862890 22025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:16.863664 22025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [term 1 LEADER]: Becoming Leader. State: Replica: 6d134526421d4e5db983778ca31e6e18, State: Running, Role: LEADER
I20260929 15:50:16.864285 22025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d134526421d4e5db983778ca31e6e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33469 } }
I20260929 15:50:16.864991 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:16.867316 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d134526421d4e5db983778ca31e6e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d134526421d4e5db983778ca31e6e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33469 } } }
I20260929 15:50:16.867872 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:16.868541 22027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d134526421d4e5db983778ca31e6e18. Latest consensus state: current_term: 1 leader_uuid: "6d134526421d4e5db983778ca31e6e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d134526421d4e5db983778ca31e6e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33469 } } }
I20260929 15:50:16.868886 22027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:16.868942 22032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:16.879261 22032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:16.890486 22032 catalog_manager.cc:1347] Generated new cluster ID: ec1e9289d5d54234b7fb646dd4f56a50
I20260929 15:50:16.890628 22032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:16.910266 22032 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:16.911963 22032 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:16.924554 22032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d134526421d4e5db983778ca31e6e18: Generated new TSK 0
I20260929 15:50:16.925568 22032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:16.939999 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21948
W20260929 15:50:16.941716 22010 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:45753
W20260929 15:50:16.942687 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 21948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:16.942855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 21948
W20260929 15:50:16.961050 21933 connection.cc:535] client connection to 127.0.0.1:33469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33469): Connection reset by peer (error 104)
2026-09-29T15:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:17 dist-test-slave-7491 krb5kdc[22046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:17 dist-test-slave-7491 krb5kdc[22046](info): set up 2 sockets
Sep 29 15:50:17 dist-test-slave-7491 krb5kdc[22046](info): commencing operation
krb5kdc: starting...
W20260929 15:50:18.061107 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:18 dist-test-slave-7491 krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 15:50:18.349050 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:18.700981 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:18.701325 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:18.701499 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:18.709589 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:18.709734 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:18.709782 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:18.709887 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:18.709971 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:18.720685 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:18.722167 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:18.723587 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:18.737695 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:18.737617 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:18.737658 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:18.739205 22059 server_base.cc:1034] running on GCE node
I20260929 15:50:18.740152 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:18.741943 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:18.743276 22059 hybrid_clock.cc:648] HybridClock initialized: now 1790697018743218 us; error 44 us; skew 500 ppm
I20260929 15:50:18.743716 22059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:18 dist-test-slave-7491 krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:18.754628 22059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 15:50:18.756516 22059 webserver.cc:460] Webserver started at http://127.0.0.1:34079/ using document root <none> and password file <none>
I20260929 15:50:18.757329 22059 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:18.757496 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:18.757968 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:18.761361 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "c3227b5a737e4b96a87e11c046772e99"
format_stamp: "Formatted at 2026-09-29 15:50:18 on dist-test-slave-7491"
server_key: "aeea404e4c6dadd0d526ee5c3a43813e"
server_key_iv: "552b0a104e692531480a02e6669133c6"
server_key_version: "encryptionkey@0"
I20260929 15:50:18.762205 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "c3227b5a737e4b96a87e11c046772e99"
format_stamp: "Formatted at 2026-09-29 15:50:18 on dist-test-slave-7491"
server_key: "aeea404e4c6dadd0d526ee5c3a43813e"
server_key_iv: "552b0a104e692531480a02e6669133c6"
server_key_version: "encryptionkey@0"
I20260929 15:50:18.768600 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:50:18.774063 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:18.776095 22059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:50:18.776408 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "c3227b5a737e4b96a87e11c046772e99"
format_stamp: "Formatted at 2026-09-29 15:50:18 on dist-test-slave-7491"
server_key: "aeea404e4c6dadd0d526ee5c3a43813e"
server_key_iv: "552b0a104e692531480a02e6669133c6"
server_key_version: "encryptionkey@0"
I20260929 15:50:18.776671 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:18.783360 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:18.789449 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:18.789812 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:18.812446 22059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43235
I20260929 15:50:18.812476 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43235 every 8 connection(s)
I20260929 15:50:18.814031 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:18.818506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22059
I20260929 15:50:18.818763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:18.819252 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:18.819401 388 external_mini_cluster.cc:1428] Setting key 84c06a64664787faff0cc4761069ab14
I20260929 15:50:18.829687 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:18.821893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:55145 (local address 127.0.0.1:43235)
0929 15:50:18.823003 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:18.823016 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:18.823067 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:50:18.825649 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:18.825680 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:18.826144 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:18.826641 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:18.827862 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:18.827869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:18.827884 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:18.827903 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:18.827970 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:18.827999 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:18.828296 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:18.828451 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 15:50:18.828665 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":419,"threads_started":1}
I20260929 15:50:18.834677 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:18.836230 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99: Bootstrap starting.
I20260929 15:50:18.841183 22126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:18.842665 22126 log.cc:824] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:18.843122 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:18.840087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:52791 (local address 127.0.0.1:43235)
0929 15:50:18.840319 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:18.840327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:18.840704 (+ 377us) server_negotiation.cc:407] Connection header received
0929 15:50:18.840920 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:18.840928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:18.840987 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:18.841109 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:18.842199 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:18.842204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:18.842207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:18.842226 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:18.842267 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:18.842283 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:18.842440 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:18.842542 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 15:50:18.842614 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:50:18.846271 22126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99: No bootstrap required, opened a new log
I20260929 15:50:18.856247 22126 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3227b5a737e4b96a87e11c046772e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260929 15:50:18.856716 22126 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:18.857007 22126 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3227b5a737e4b96a87e11c046772e99, State: Initialized, Role: FOLLOWER
I20260929 15:50:18.858253 22126 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3227b5a737e4b96a87e11c046772e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260929 15:50:18.858582 22126 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:18.858760 22126 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:18.859244 22126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:18.863102 22126 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3227b5a737e4b96a87e11c046772e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260929 15:50:18.863744 22126 leader_election.cc:302] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3227b5a737e4b96a87e11c046772e99; no voters:
I20260929 15:50:18.864492 22126 leader_election.cc:288] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:18.864639 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:18.865415 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [term 1 LEADER]: Becoming Leader. State: Replica: c3227b5a737e4b96a87e11c046772e99, State: Running, Role: LEADER
I20260929 15:50:18.865978 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3227b5a737e4b96a87e11c046772e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } }
I20260929 15:50:18.866860 22126 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:18.869007 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3227b5a737e4b96a87e11c046772e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3227b5a737e4b96a87e11c046772e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } } }
I20260929 15:50:18.869477 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:18.869992 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3227b5a737e4b96a87e11c046772e99. Latest consensus state: current_term: 1 leader_uuid: "c3227b5a737e4b96a87e11c046772e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3227b5a737e4b96a87e11c046772e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43235 } } }
I20260929 15:50:18.870489 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:18.873672 22142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:18.881343 22142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:18.892568 22142 catalog_manager.cc:1347] Generated new cluster ID: b31e9228e1fd45ae82dea9f3a59e657b
I20260929 15:50:18.892722 22142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:18.908730 22142 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:18.909272 22142 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:18.917186 22142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3227b5a737e4b96a87e11c046772e99: Generated new TSK 0
I20260929 15:50:18.918604 22142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:18.926087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22059
W20260929 15:50:18.927304 22121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.96:55145
W20260929 15:50:18.928094 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22059: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:18.928297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22059
2026-09-29T15:50:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:19 dist-test-slave-7491 krb5kdc[22157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:19 dist-test-slave-7491 krb5kdc[22157](info): set up 2 sockets
Sep 29 15:50:19 dist-test-slave-7491 krb5kdc[22157](info): commencing operation
krb5kdc: starting...
W20260929 15:50:20.096992 388 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.009s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:20 dist-test-slave-7491 krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:20.381045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40895
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:20.759006 22170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:20.759769 22170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:20.760032 22170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:20.768034 22170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:20.768285 22170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:20.768410 22170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:20.768502 22170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:20.768592 22170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:20.780038 22170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:20.781589 22170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:20.783007 22170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:20.795742 22175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:20.796689 22176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:20.796830 22178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:20.797432 22170 server_base.cc:1034] running on GCE node
I20260929 15:50:20.798628 22170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:20.800644 22170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:20.801949 22170 hybrid_clock.cc:648] HybridClock initialized: now 1790697020801928 us; error 90 us; skew 500 ppm
I20260929 15:50:20.802402 22170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:20 dist-test-slave-7491 krb5kdc[22157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:20.806084 22170 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:20.807804 22170 webserver.cc:460] Webserver started at http://127.0.97.62:44965/ using document root <none> and password file <none>
I20260929 15:50:20.808634 22170 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:20.808751 22170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:20.809008 22170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:20.812391 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "3c4a65330dc6497a84ffd30a31cf3164"
format_stamp: "Formatted at 2026-09-29 15:50:20 on dist-test-slave-7491"
server_key: "a3f6e621d64f77346b1af232c9c36fd4"
server_key_iv: "dee54b4b8a979e2e3800015651aae553"
server_key_version: "encryptionkey@0"
I20260929 15:50:20.813215 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c4a65330dc6497a84ffd30a31cf3164"
format_stamp: "Formatted at 2026-09-29 15:50:20 on dist-test-slave-7491"
server_key: "a3f6e621d64f77346b1af232c9c36fd4"
server_key_iv: "dee54b4b8a979e2e3800015651aae553"
server_key_version: "encryptionkey@0"
I20260929 15:50:20.818979 22170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:50:20.823298 22185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:20.825063 22170 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:50:20.825384 22170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "3c4a65330dc6497a84ffd30a31cf3164"
format_stamp: "Formatted at 2026-09-29 15:50:20 on dist-test-slave-7491"
server_key: "a3f6e621d64f77346b1af232c9c36fd4"
server_key_iv: "dee54b4b8a979e2e3800015651aae553"
server_key_version: "encryptionkey@0"
I20260929 15:50:20.825629 22170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:20.852833 22170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:20.858913 22170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:20.859253 22170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:20.882630 22170 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40895
I20260929 15:50:20.882648 22236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40895 every 8 connection(s)
I20260929 15:50:20.884236 22170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:20.888831 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:20.891371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22170
I20260929 15:50:20.891577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:20.892536 388 external_mini_cluster.cc:1428] Setting key 89dccc0bfc655d1e4130d818e3e945fe
I20260929 15:50:20.903131 22237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164: Bootstrap starting.
I20260929 15:50:20.906661 22237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:20.907797 22237 log.cc:824] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:50:20 dist-test-slave-7491 krb5kdc[22157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:50:20.911682 22237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164: No bootstrap required, opened a new log
I20260929 15:50:20.919862 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:20.895625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41363 (local address 127.0.97.62:40895)
0929 15:50:20.896666 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:20.896680 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:20.896729 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:50:20.899100 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:20.899131 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:20.899624 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:20.900111 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:20.901736 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:20.903632 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:20.905160 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:20.905907 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:20.906124 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:20.911508 (+ 5384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:20.911517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:20.911533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:20.911580 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:20.915051 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:20.916099 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:20.916106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:20.916117 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:20.916382 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:20.916901 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:20.916905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:20.916907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:20.917511 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:20.917890 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:20.918393 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 15:50:20.918741 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":479,"threads_started":1}
I20260929 15:50:20.920434 22237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a65330dc6497a84ffd30a31cf3164" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40895 } }
I20260929 15:50:20.920867 22237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:20.920997 22237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4a65330dc6497a84ffd30a31cf3164, State: Initialized, Role: FOLLOWER
I20260929 15:50:20.921708 22237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a65330dc6497a84ffd30a31cf3164" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40895 } }
I20260929 15:50:20.922000 22237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:20.922159 22237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:20.922343 22237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:20.924613 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:20.925225 22237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a65330dc6497a84ffd30a31cf3164" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40895 } }
I20260929 15:50:20.925822 22237 leader_election.cc:302] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c4a65330dc6497a84ffd30a31cf3164; no voters:
I20260929 15:50:20.926494 22237 leader_election.cc:288] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:20.926668 22242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:20.928809 22237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:20.932924 22242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [term 1 LEADER]: Becoming Leader. State: Replica: 3c4a65330dc6497a84ffd30a31cf3164, State: Running, Role: LEADER
I20260929 15:50:20.933948 22242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a65330dc6497a84ffd30a31cf3164" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40895 } }
I20260929 15:50:20.938060 22244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c4a65330dc6497a84ffd30a31cf3164. Latest consensus state: current_term: 1 leader_uuid: "3c4a65330dc6497a84ffd30a31cf3164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a65330dc6497a84ffd30a31cf3164" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40895 } } }
I20260929 15:50:20.938043 22243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c4a65330dc6497a84ffd30a31cf3164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c4a65330dc6497a84ffd30a31cf3164" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40895 } } }
I20260929 15:50:20.940222 22244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:20.940522 22243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:20.942140 22253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:20.951100 22253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:20.962579 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:20.941486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49315 (local address 127.0.97.62:40895)
0929 15:50:20.941738 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:20.941752 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:20.942851 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 15:50:20.943191 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:20.943206 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:20.943320 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:20.943412 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:20.945602 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:20.946931 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:20.949096 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:20.949845 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:20.949967 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:20.953812 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:20.953820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:20.953826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:20.953882 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:20.958439 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:20.959727 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:20.959732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:20.959736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:20.959898 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:20.960565 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:20.960573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:20.960576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:20.960959 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:20.961329 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:20.961971 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 15:50:20.962198 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:50:20.963739 22253 catalog_manager.cc:1347] Generated new cluster ID: a3b47019a4b746e4899bf443825c9bdd
I20260929 15:50:20.963937 22253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:20.977591 22253 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:20.979048 22253 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:20.986950 22253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c4a65330dc6497a84ffd30a31cf3164: Generated new TSK 0
I20260929 15:50:20.988015 22253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:20.999152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22170
I20260929 15:50:21.000936 22232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:21.098073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22170
W20260929 15:50:21.118145 22155 connection.cc:535] client connection to 127.0.97.62:40895 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40895): Connection reset by peer (error 104)
2026-09-29T15:50:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2170 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:21 dist-test-slave-7491 krb5kdc[22269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:21 dist-test-slave-7491 krb5kdc[22269](info): set up 2 sockets
Sep 29 15:50:21 dist-test-slave-7491 krb5kdc[22269](info): commencing operation
krb5kdc: starting...
W20260929 15:50:22.240875 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:22 dist-test-slave-7491 krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:22.540412 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41987
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:22.940434 22282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:22.940850 22282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:22.941046 22282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:22.949898 22282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:22.950330 22282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:22.950600 22282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:22.950766 22282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:22.950863 22282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:22.962743 22282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:22.965116 22282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:22.967043 22282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:22.981431 22287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:22.981675 22290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:22.981855 22288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:22.982558 22282 server_base.cc:1034] running on GCE node
I20260929 15:50:22.983376 22282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:22.985213 22282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:22.986487 22282 hybrid_clock.cc:648] HybridClock initialized: now 1790697022986435 us; error 137 us; skew 500 ppm
I20260929 15:50:22.986931 22282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:22 dist-test-slave-7491 krb5kdc[22269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:22.990567 22282 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:22.992707 22282 webserver.cc:460] Webserver started at http://127.0.97.62:32983/ using document root <none> and password file <none>
I20260929 15:50:22.993541 22282 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:22.993714 22282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:22.994088 22282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:22.997550 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "dd23972e63144b54878889cb72499221"
format_stamp: "Formatted at 2026-09-29 15:50:22 on dist-test-slave-7491"
server_key: "e65af7c2e4243190f253a8a78309c0c1"
server_key_iv: "ba55ae5eead6ce37983ea2260cf53651"
server_key_version: "encryptionkey@0"
I20260929 15:50:22.998627 22282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd23972e63144b54878889cb72499221"
format_stamp: "Formatted at 2026-09-29 15:50:22 on dist-test-slave-7491"
server_key: "e65af7c2e4243190f253a8a78309c0c1"
server_key_iv: "ba55ae5eead6ce37983ea2260cf53651"
server_key_version: "encryptionkey@0"
I20260929 15:50:23.006217 22282 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:50:23.010708 22297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:23.013044 22282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:50:23.013283 22282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "dd23972e63144b54878889cb72499221"
format_stamp: "Formatted at 2026-09-29 15:50:22 on dist-test-slave-7491"
server_key: "e65af7c2e4243190f253a8a78309c0c1"
server_key_iv: "ba55ae5eead6ce37983ea2260cf53651"
server_key_version: "encryptionkey@0"
I20260929 15:50:23.013545 22282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:23.032577 22282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:23.039290 22282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:23.039657 22282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:23.062780 22282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41987
I20260929 15:50:23.062808 22348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41987 every 8 connection(s)
I20260929 15:50:23.064371 22282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:23.068691 22349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:23.073006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22282
I20260929 15:50:23.073217 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:23.073848 388 external_mini_cluster.cc:1428] Setting key cc70dde8ce0e1bbad879828da923eaeb
I20260929 15:50:23.084759 22349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221: Bootstrap starting.
I20260929 15:50:23.089026 22349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:23.090412 22349 log.cc:824] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:23.092545 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:23.076501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34801 (local address 127.0.97.62:41987)
0929 15:50:23.077740 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:23.077753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:23.077807 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:50:23.080032 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:23.080078 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:23.080990 (+ 912us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:23.081425 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:23.082086 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:23.083483 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:23.085571 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:23.086587 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:23.086811 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:23.088507 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:23.088517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:23.088536 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:23.089224 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:23.089341 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:23.089380 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:23.089561 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:23.090037 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 15:50:23.090267 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":657,"threads_started":1}
I20260929 15:50:23.095057 22349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221: No bootstrap required, opened a new log
I20260929 15:50:23.098623 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:23.105501 22349 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23972e63144b54878889cb72499221" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 15:50:23.106051 22349 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:23.106195 22349 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd23972e63144b54878889cb72499221, State: Initialized, Role: FOLLOWER
I20260929 15:50:23.106917 22349 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23972e63144b54878889cb72499221" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 15:50:23.107195 22349 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:23.107349 22349 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:23.107559 22349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:23.110177 22349 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23972e63144b54878889cb72499221" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 15:50:23.111003 22349 leader_election.cc:302] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd23972e63144b54878889cb72499221; no voters:
I20260929 15:50:23.111581 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:23.104071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49153 (local address 127.0.97.62:41987)
0929 15:50:23.104324 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:23.104332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:23.104834 (+ 502us) server_negotiation.cc:407] Connection header received
0929 15:50:23.105072 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:23.105077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:23.105188 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:23.105312 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:23.105881 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:23.106901 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:23.108654 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:23.109195 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:23.109347 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:23.110626 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:23.110631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:23.110635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:23.110655 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:23.110694 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:23.110711 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:23.110965 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:23.111160 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 15:50:23.111217 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 15:50:23.111866 22349 leader_election.cc:288] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:23.111984 22359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:23.115243 22349 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:23.116636 22359 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [term 1 LEADER]: Becoming Leader. State: Replica: dd23972e63144b54878889cb72499221, State: Running, Role: LEADER
I20260929 15:50:23.117604 22359 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23972e63144b54878889cb72499221" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 15:50:23.121734 22361 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd23972e63144b54878889cb72499221. Latest consensus state: current_term: 1 leader_uuid: "dd23972e63144b54878889cb72499221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23972e63144b54878889cb72499221" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } } }
I20260929 15:50:23.122387 22361 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:23.123103 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd23972e63144b54878889cb72499221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd23972e63144b54878889cb72499221" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } } }
I20260929 15:50:23.123565 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:23.128721 22367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:23.133473 22367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:23.143388 22367 catalog_manager.cc:1347] Generated new cluster ID: c15f1d87bc68451e8872b479f45b97e7
I20260929 15:50:23.143549 22367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:23.175489 22367 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:23.177086 22367 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:23.187100 22367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd23972e63144b54878889cb72499221: Generated new TSK 0
I20260929 15:50:23.188082 22367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:23.203152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22282
W20260929 15:50:23.204668 22344 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34801
W20260929 15:50:23.205629 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:23.205807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22282
W20260929 15:50:23.225198 22267 connection.cc:535] client connection to 127.0.97.62:41987 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41987): Connection reset by peer (error 104)
2026-09-29T15:50:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:23 dist-test-slave-7491 krb5kdc[22381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:23 dist-test-slave-7491 krb5kdc[22381](info): set up 2 sockets
Sep 29 15:50:23 dist-test-slave-7491 krb5kdc[22381](info): commencing operation
krb5kdc: starting...
W20260929 15:50:24.361095 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:24 dist-test-slave-7491 krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:24.672477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42265
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:25.052152 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:25.052594 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:25.052695 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:25.061309 22394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:25.061453 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:25.061501 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:25.061554 22394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:25.061601 22394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:25.072602 22394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:25.073999 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:25.075331 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:25.088485 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:25.088393 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:25.088505 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:25.090049 22394 server_base.cc:1034] running on GCE node
I20260929 15:50:25.091033 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:25.092754 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:25.094028 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790697025093983 us; error 39 us; skew 500 ppm
I20260929 15:50:25.094391 22394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:25 dist-test-slave-7491 krb5kdc[22381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:25.102063 22394 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:25.104524 22394 webserver.cc:460] Webserver started at http://127.0.97.62:33867/ using document root <none> and password file <none>
I20260929 15:50:25.105297 22394 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:25.105424 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:25.105688 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:25.109009 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec"
format_stamp: "Formatted at 2026-09-29 15:50:25 on dist-test-slave-7491"
server_key: "2f30884af792094681041036fb384655"
server_key_iv: "8f54611a018f0cbf7fd0a20bd9c658dd"
server_key_version: "encryptionkey@0"
I20260929 15:50:25.109784 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec"
format_stamp: "Formatted at 2026-09-29 15:50:25 on dist-test-slave-7491"
server_key: "2f30884af792094681041036fb384655"
server_key_iv: "8f54611a018f0cbf7fd0a20bd9c658dd"
server_key_version: "encryptionkey@0"
I20260929 15:50:25.116287 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:50:25.120630 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:25.122318 22394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 15:50:25.122601 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec"
format_stamp: "Formatted at 2026-09-29 15:50:25 on dist-test-slave-7491"
server_key: "2f30884af792094681041036fb384655"
server_key_iv: "8f54611a018f0cbf7fd0a20bd9c658dd"
server_key_version: "encryptionkey@0"
I20260929 15:50:25.122937 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:25.129616 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:25.135064 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:25.135432 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:25.157240 22394 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42265
I20260929 15:50:25.157258 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42265 every 8 connection(s)
I20260929 15:50:25.158855 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:25.162971 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:25.163199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22394
I20260929 15:50:25.163360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:25.163821 388 external_mini_cluster.cc:1428] Setting key 051aa260ddb8236cab2e3a1cd1126c7f
I20260929 15:50:25.172302 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:25.165627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37871 (local address 127.0.97.62:42265)
0929 15:50:25.166584 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:25.166596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:25.166645 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:50:25.168358 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:25.168383 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:25.168797 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:25.169185 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:25.170431 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:25.170438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:25.170455 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:25.170473 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:25.170545 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:25.170572 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:25.170786 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:25.170915 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 15:50:25.171165 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":448,"threads_started":1}
I20260929 15:50:25.177049 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:25.178434 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec: Bootstrap starting.
I20260929 15:50:25.183096 22461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:25.184422 22461 log.cc:824] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:25.184859 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:25.182109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42789 (local address 127.0.97.62:42265)
0929 15:50:25.182312 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:25.182321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:25.182716 (+ 395us) server_negotiation.cc:407] Connection header received
0929 15:50:25.182938 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:25.182948 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:25.183009 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:25.183137 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:25.184113 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:25.184118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:25.184123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:25.184154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:25.184253 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:25.184281 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:25.184420 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:25.184501 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 15:50:25.184569 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:50:25.188277 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec: No bootstrap required, opened a new log
I20260929 15:50:25.196727 22461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42265 } }
I20260929 15:50:25.197314 22461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:25.197432 22461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3b9a54bdc64e9499eb05f0e471e3ec, State: Initialized, Role: FOLLOWER
I20260929 15:50:25.198434 22461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42265 } }
I20260929 15:50:25.198969 22461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:25.199167 22461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:25.199365 22461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:25.202081 22461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42265 } }
I20260929 15:50:25.202723 22461 leader_election.cc:302] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e3b9a54bdc64e9499eb05f0e471e3ec; no voters:
I20260929 15:50:25.203382 22461 leader_election.cc:288] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:25.203610 22471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:25.204358 22471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [term 1 LEADER]: Becoming Leader. State: Replica: 6e3b9a54bdc64e9499eb05f0e471e3ec, State: Running, Role: LEADER
I20260929 15:50:25.204943 22471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42265 } }
I20260929 15:50:25.205598 22461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:25.208220 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42265 } } }
I20260929 15:50:25.208145 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e3b9a54bdc64e9499eb05f0e471e3ec. Latest consensus state: current_term: 1 leader_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3b9a54bdc64e9499eb05f0e471e3ec" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42265 } } }
I20260929 15:50:25.208673 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:25.208688 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:25.211898 22479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:25.218189 22479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:25.228540 22479 catalog_manager.cc:1347] Generated new cluster ID: c8b3729368b341458899eb8b322d30ef
I20260929 15:50:25.228684 22479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:25.246796 22479 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:25.247400 22479 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:25.255782 22479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e3b9a54bdc64e9499eb05f0e471e3ec: Generated new TSK 0
I20260929 15:50:25.257004 22479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:25.263778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22394
W20260929 15:50:25.264973 22456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37871
W20260929 15:50:25.265740 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22394: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:25.265914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22394
2026-09-29T15:50:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2055 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:25 dist-test-slave-7491 krb5kdc[22492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:25 dist-test-slave-7491 krb5kdc[22492](info): set up 2 sockets
Sep 29 15:50:25 dist-test-slave-7491 krb5kdc[22492](info): commencing operation
krb5kdc: starting...
W20260929 15:50:26.412827 388 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:26 dist-test-slave-7491 krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:26.697139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32815
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:27.075265 22505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:27.075595 22505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:27.075695 22505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:27.083279 22505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:27.083419 22505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:27.083467 22505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:27.083595 22505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:27.083666 22505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:27.094076 22505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:27.095526 22505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:27.097059 22505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:27.109937 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:27.109957 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:27.110036 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:27.110926 22505 server_base.cc:1034] running on GCE node
I20260929 15:50:27.111763 22505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:27.113585 22505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:27.114931 22505 hybrid_clock.cc:648] HybridClock initialized: now 1790697027114876 us; error 56 us; skew 500 ppm
I20260929 15:50:27.115378 22505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:27 dist-test-slave-7491 krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:27.119036 22505 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:27.121066 22505 webserver.cc:460] Webserver started at http://127.0.97.62:36751/ using document root <none> and password file <none>
I20260929 15:50:27.121891 22505 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:27.122071 22505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:27.122429 22505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:27.125991 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "8aec173ff8bb4f7a80e49512bcf6482a"
format_stamp: "Formatted at 2026-09-29 15:50:27 on dist-test-slave-7491"
server_key: "9460e788c38c2010577e92a9d2b851b3"
server_key_iv: "bfbe973cb4d8b13062111dd8833d3021"
server_key_version: "encryptionkey@0"
I20260929 15:50:27.126924 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "8aec173ff8bb4f7a80e49512bcf6482a"
format_stamp: "Formatted at 2026-09-29 15:50:27 on dist-test-slave-7491"
server_key: "9460e788c38c2010577e92a9d2b851b3"
server_key_iv: "bfbe973cb4d8b13062111dd8833d3021"
server_key_version: "encryptionkey@0"
I20260929 15:50:27.133270 22505 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:50:27.137204 22520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:27.139353 22505 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:50:27.139605 22505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "8aec173ff8bb4f7a80e49512bcf6482a"
format_stamp: "Formatted at 2026-09-29 15:50:27 on dist-test-slave-7491"
server_key: "9460e788c38c2010577e92a9d2b851b3"
server_key_iv: "bfbe973cb4d8b13062111dd8833d3021"
server_key_version: "encryptionkey@0"
I20260929 15:50:27.139860 22505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:27.166626 22505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:27.171367 22505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:27.171710 22505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:27.192580 22505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32815
I20260929 15:50:27.192624 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32815 every 8 connection(s)
I20260929 15:50:27.194048 22505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:27.197880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22505
I20260929 15:50:27.198072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:27.198203 22572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:27.198607 388 external_mini_cluster.cc:1428] Setting key be4acda2e9a60a3a7d54b883f8927b99
I20260929 15:50:27.212648 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a: Bootstrap starting.
Sep 29 15:50:27 dist-test-slave-7491 krb5kdc[22492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:50:27.216223 22572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:27.217744 22572 log.cc:824] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:27.221570 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a: No bootstrap required, opened a new log
I20260929 15:50:27.224392 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:27.200956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53261 (local address 127.0.97.62:32815)
0929 15:50:27.202065 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:27.202079 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:27.202131 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:50:27.204347 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:27.204377 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:27.204890 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:27.205357 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:27.206903 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:27.208319 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:27.209678 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:27.210461 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:27.210688 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:27.215498 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:27.215507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:27.215524 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:27.215571 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:27.219166 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:27.220037 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:27.220047 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:27.220055 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:27.220322 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:27.220760 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:27.220765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:27.220768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:27.221501 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:27.221820 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:27.222386 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 15:50:27.222886 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":631,"threads_started":1}
I20260929 15:50:27.229305 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:27.231455 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32815 } }
I20260929 15:50:27.232122 22572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:27.232327 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aec173ff8bb4f7a80e49512bcf6482a, State: Initialized, Role: FOLLOWER
I20260929 15:50:27.233047 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32815 } }
I20260929 15:50:27.233336 22572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:27.233506 22572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:27.233688 22572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:27.236714 22572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32815 } }
I20260929 15:50:27.237347 22572 leader_election.cc:302] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aec173ff8bb4f7a80e49512bcf6482a; no voters:
I20260929 15:50:27.238044 22572 leader_election.cc:288] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:27.238193 22582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:27.240471 22582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [term 1 LEADER]: Becoming Leader. State: Replica: 8aec173ff8bb4f7a80e49512bcf6482a, State: Running, Role: LEADER
I20260929 15:50:27.241226 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:27.241226 22582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32815 } }
I20260929 15:50:27.244868 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32815 } } }
I20260929 15:50:27.245406 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:27.250622 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aec173ff8bb4f7a80e49512bcf6482a. Latest consensus state: current_term: 1 leader_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec173ff8bb4f7a80e49512bcf6482a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32815 } } }
I20260929 15:50:27.251864 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:27.256417 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:27.234743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60455 (local address 127.0.97.62:32815)
0929 15:50:27.235000 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:27.235009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:27.235753 (+ 744us) server_negotiation.cc:407] Connection header received
0929 15:50:27.235911 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:27.235917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:27.236003 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:27.236123 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:27.238191 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:27.239845 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:27.241619 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:27.242109 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:27.242274 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:27.245738 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:27.245747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:27.245752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:27.245796 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:27.250806 (+ 5010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:27.252209 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:27.252219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:27.252225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:27.252475 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:27.253073 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:27.253082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:27.253087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:27.253624 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:27.253965 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:27.255656 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0929 15:50:27.256003 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:50:27.257822 22588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:27.265126 22588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:27.277925 22588 catalog_manager.cc:1347] Generated new cluster ID: 89319c876d7d44b7b3769dbb5a70703a
I20260929 15:50:27.278156 22588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:27.297672 22588 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:27.299734 22588 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:27.311146 22588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aec173ff8bb4f7a80e49512bcf6482a: Generated new TSK 0
I20260929 15:50:27.313346 22588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:27.320921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22505
I20260929 15:50:27.322156 22567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:27.417755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22505
2026-09-29T15:50:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:27 dist-test-slave-7491 krb5kdc[22604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:27 dist-test-slave-7491 krb5kdc[22604](info): set up 2 sockets
Sep 29 15:50:27 dist-test-slave-7491 krb5kdc[22604](info): commencing operation
krb5kdc: starting...
W20260929 15:50:28.573101 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:28 dist-test-slave-7491 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:28.875653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35093
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:29.258836 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:29.259186 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:29.259303 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:29.268790 22617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:29.268949 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:29.269001 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:29.269059 22617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:29.269106 22617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:29.281430 22617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:29.282975 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:29.284549 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:29.299404 22625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:29.299456 22623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:29.299491 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:29.299999 22617 server_base.cc:1034] running on GCE node
I20260929 15:50:29.301457 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:29.303656 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:29.304929 22617 hybrid_clock.cc:648] HybridClock initialized: now 1790697029304827 us; error 107 us; skew 500 ppm
I20260929 15:50:29.305392 22617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:29 dist-test-slave-7491 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:29.309068 22617 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:29.311128 22617 webserver.cc:460] Webserver started at http://127.0.97.62:38069/ using document root <none> and password file <none>
I20260929 15:50:29.311970 22617 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:29.312107 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:29.312551 22617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:29.316314 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "efdaf7f64dc7484aa6c00ce971ea1796"
format_stamp: "Formatted at 2026-09-29 15:50:29 on dist-test-slave-7491"
server_key: "2d3ae0f2612f3d93fc1000044deddb3e"
server_key_iv: "0787a959c8b0ff6fcc4707a32ff97f18"
server_key_version: "encryptionkey@0"
I20260929 15:50:29.317270 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "efdaf7f64dc7484aa6c00ce971ea1796"
format_stamp: "Formatted at 2026-09-29 15:50:29 on dist-test-slave-7491"
server_key: "2d3ae0f2612f3d93fc1000044deddb3e"
server_key_iv: "0787a959c8b0ff6fcc4707a32ff97f18"
server_key_version: "encryptionkey@0"
I20260929 15:50:29.324864 22617 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 15:50:29.329466 22632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:29.331611 22617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 15:50:29.331907 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "efdaf7f64dc7484aa6c00ce971ea1796"
format_stamp: "Formatted at 2026-09-29 15:50:29 on dist-test-slave-7491"
server_key: "2d3ae0f2612f3d93fc1000044deddb3e"
server_key_iv: "0787a959c8b0ff6fcc4707a32ff97f18"
server_key_version: "encryptionkey@0"
I20260929 15:50:29.332156 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:29.354375 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:29.359578 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:29.359917 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:29.380671 22617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35093
I20260929 15:50:29.380722 22683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35093 every 8 connection(s)
I20260929 15:50:29.382124 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:29.386261 22684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:29.390331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22617
I20260929 15:50:29.390614 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:29.391309 388 external_mini_cluster.cc:1428] Setting key 0710cad84b0517b9d63a2a2e67c7f114
I20260929 15:50:29.400880 22684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796: Bootstrap starting.
I20260929 15:50:29.405053 22684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:29.406291 22684 log.cc:824] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:29.406338 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:29.393579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59661 (local address 127.0.97.62:35093)
0929 15:50:29.394450 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:29.394463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:29.394511 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:50:29.396364 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:29.396392 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:29.396905 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:29.397287 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:29.398013 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:29.399430 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:29.401451 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:29.402169 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:29.402313 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:29.403719 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:29.403727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:29.403746 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:29.404432 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:29.404520 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:29.404547 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:29.404646 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:29.404960 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 15:50:29.405312 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":396,"threads_started":1}
I20260929 15:50:29.409785 22684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796: No bootstrap required, opened a new log
I20260929 15:50:29.410817 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:29.419415 22684 raft_consensus.cc:348] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35093 } }
I20260929 15:50:29.420006 22684 raft_consensus.cc:374] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:29.420114 22684 raft_consensus.cc:729] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdaf7f64dc7484aa6c00ce971ea1796, State: Initialized, Role: FOLLOWER
I20260929 15:50:29.420920 22684 consensus_queue.cc:260] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35093 } }
I20260929 15:50:29.421178 22684 raft_consensus.cc:388] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:29.421404 22684 raft_consensus.cc:482] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:29.421590 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:29.423856 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:29.416301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34593 (local address 127.0.97.62:35093)
0929 15:50:29.416665 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:29.416672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:29.416995 (+ 323us) server_negotiation.cc:407] Connection header received
0929 15:50:29.417183 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:29.417188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:29.417261 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:29.417366 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:29.418036 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:29.418827 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:29.420870 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:29.421476 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:29.421623 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:29.422889 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:29.422894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:29.422898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:29.422923 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:29.422967 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:29.422984 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:29.423077 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:29.423281 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 15:50:29.423377 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 15:50:29.425043 22684 raft_consensus.cc:504] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35093 } }
I20260929 15:50:29.425691 22684 leader_election.cc:302] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efdaf7f64dc7484aa6c00ce971ea1796; no voters:
I20260929 15:50:29.426332 22684 leader_election.cc:288] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:29.426502 22694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:29.427215 22694 raft_consensus.cc:686] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [term 1 LEADER]: Becoming Leader. State: Replica: efdaf7f64dc7484aa6c00ce971ea1796, State: Running, Role: LEADER
I20260929 15:50:29.427870 22694 consensus_queue.cc:237] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35093 } }
I20260929 15:50:29.428658 22684 sys_catalog.cc:564] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:29.431036 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35093 } } }
I20260929 15:50:29.432528 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efdaf7f64dc7484aa6c00ce971ea1796. Latest consensus state: current_term: 1 leader_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdaf7f64dc7484aa6c00ce971ea1796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35093 } } }
I20260929 15:50:29.433254 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:29.433545 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796 [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:29.435235 22703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:29.441855 22703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:29.452870 22703 catalog_manager.cc:1347] Generated new cluster ID: 708c4b4280c5482197d7c61381bf7a7d
I20260929 15:50:29.453024 22703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:29.467530 22703 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:29.469000 22703 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:29.482687 22703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efdaf7f64dc7484aa6c00ce971ea1796: Generated new TSK 0
I20260929 15:50:29.483672 22703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:29.493860 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22617
W20260929 15:50:29.495358 22679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59661
W20260929 15:50:29.496317 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:29.496495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22617
2026-09-29T15:50:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:29 dist-test-slave-7491 krb5kdc[22715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:29 dist-test-slave-7491 krb5kdc[22715](info): set up 2 sockets
Sep 29 15:50:29 dist-test-slave-7491 krb5kdc[22715](info): commencing operation
krb5kdc: starting...
W20260929 15:50:30.652793 388 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:30 dist-test-slave-7491 krb5kdc[22715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:30.940024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46071
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:31.306919 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:31.307262 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:31.307412 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:31.315155 22728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:31.315296 22728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:31.315344 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:31.315464 22728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:31.315577 22728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:31.326164 22728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:31.327658 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:31.329377 22728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:31.343497 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:31.343528 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:31.343947 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:31.344385 22728 server_base.cc:1034] running on GCE node
I20260929 15:50:31.345319 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:31.346953 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:31.348320 22728 hybrid_clock.cc:648] HybridClock initialized: now 1790697031348252 us; error 56 us; skew 500 ppm
I20260929 15:50:31.348745 22728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:31 dist-test-slave-7491 krb5kdc[22715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:31.358741 22728 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:31.361018 22728 webserver.cc:460] Webserver started at http://127.0.97.62:42933/ using document root <none> and password file <none>
I20260929 15:50:31.361825 22728 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:31.361959 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:31.362242 22728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:31.365593 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "d1d2fe3f0004452a9f023bd6a25518ce"
format_stamp: "Formatted at 2026-09-29 15:50:31 on dist-test-slave-7491"
server_key: "c0b378dcf2c24f475c5b71fd01c0947e"
server_key_iv: "51a35a8bb1d250334d8d082d9133f9f9"
server_key_version: "encryptionkey@0"
I20260929 15:50:31.366434 22728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1d2fe3f0004452a9f023bd6a25518ce"
format_stamp: "Formatted at 2026-09-29 15:50:31 on dist-test-slave-7491"
server_key: "c0b378dcf2c24f475c5b71fd01c0947e"
server_key_iv: "51a35a8bb1d250334d8d082d9133f9f9"
server_key_version: "encryptionkey@0"
I20260929 15:50:31.373106 22728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:50:31.378500 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:31.380290 22728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260929 15:50:31.380467 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "d1d2fe3f0004452a9f023bd6a25518ce"
format_stamp: "Formatted at 2026-09-29 15:50:31 on dist-test-slave-7491"
server_key: "c0b378dcf2c24f475c5b71fd01c0947e"
server_key_iv: "51a35a8bb1d250334d8d082d9133f9f9"
server_key_version: "encryptionkey@0"
I20260929 15:50:31.380671 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:31.386940 22728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:31.392115 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:31.392513 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:31.415082 22728 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46071
I20260929 15:50:31.415093 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46071 every 8 connection(s)
I20260929 15:50:31.416679 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:31.420964 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:31.421167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22728
I20260929 15:50:31.421350 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:31.421898 388 external_mini_cluster.cc:1428] Setting key ea9952f6d8e8656d76715bd72beabe54
I20260929 15:50:31.430850 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:31.423977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36627 (local address 127.0.97.62:46071)
0929 15:50:31.424973 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:31.424991 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:31.425037 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:50:31.426712 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:31.426737 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:31.427142 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:31.427667 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:31.429004 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:31.429010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:31.429029 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:31.429053 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:31.429114 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:31.429137 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:31.429420 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:31.429582 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 15:50:31.429806 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":453,"threads_started":1}
I20260929 15:50:31.435887 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:31.436657 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce: Bootstrap starting.
I20260929 15:50:31.440308 22795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:31.441937 22795 log.cc:824] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:31.443447 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:31.440597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46903 (local address 127.0.97.62:46071)
0929 15:50:31.440924 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:31.440931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:31.441311 (+ 380us) server_negotiation.cc:407] Connection header received
0929 15:50:31.441475 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:31.441481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:31.441559 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:31.441698 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:31.442707 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:31.442712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:31.442720 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:50:31.442747 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:31.442796 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:50:31.442813 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:31.442952 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:31.443083 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 15:50:31.443150 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:50:31.445606 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce: No bootstrap required, opened a new log
I20260929 15:50:31.455452 22795 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46071 } }
I20260929 15:50:31.455969 22795 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:50:31.456095 22795 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d2fe3f0004452a9f023bd6a25518ce, State: Initialized, Role: FOLLOWER
I20260929 15:50:31.456868 22795 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46071 } }
I20260929 15:50:31.457176 22795 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:50:31.457350 22795 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:50:31.457566 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:31.459975 22795 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46071 } }
I20260929 15:50:31.460616 22795 leader_election.cc:302] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d2fe3f0004452a9f023bd6a25518ce; no voters:
I20260929 15:50:31.461319 22795 leader_election.cc:288] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:50:31.461508 22805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:31.462172 22805 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [term 1 LEADER]: Becoming Leader. State: Replica: d1d2fe3f0004452a9f023bd6a25518ce, State: Running, Role: LEADER
I20260929 15:50:31.462807 22805 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46071 } }
I20260929 15:50:31.463567 22795 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:31.465993 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46071 } } }
I20260929 15:50:31.466523 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:31.467049 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d2fe3f0004452a9f023bd6a25518ce. Latest consensus state: current_term: 1 leader_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d2fe3f0004452a9f023bd6a25518ce" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46071 } } }
I20260929 15:50:31.467415 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:31.467687 22813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:50:31.473488 22813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:31.483302 22813 catalog_manager.cc:1347] Generated new cluster ID: 52e8ad01ce19472680f745c3cc6ffc50
I20260929 15:50:31.483448 22813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:31.496155 22813 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:31.496770 22813 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:31.517323 22813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d2fe3f0004452a9f023bd6a25518ce: Generated new TSK 0
I20260929 15:50:31.518568 22813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:50:31.536026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22728
W20260929 15:50:31.537564 22790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36627
W20260929 15:50:31.538352 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:50:31.538539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22728
2026-09-29T15:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2042 ms)
[----------] 18 tests from AuthTokenIssuingTest (37480 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:31 dist-test-slave-7491 krb5kdc[22826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:31 dist-test-slave-7491 krb5kdc[22826](info): set up 2 sockets
Sep 29 15:50:31 dist-test-slave-7491 krb5kdc[22826](info): commencing operation
krb5kdc: starting...
W20260929 15:50:32.698042 388 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:32 dist-test-slave-7491 krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:33.004408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39637
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39637,127.0.97.61:41597,127.0.97.60:35945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:33.382975 22839 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:50:33.383458 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:33.383594 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:33.383744 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:33.391816 22839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:33.391996 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:33.392108 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:33.392271 22839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:33.392392 22839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:33.403126 22839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39637,127.0.97.61:41597,127.0.97.60:35945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:33.404691 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:33.406145 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:33.419272 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:33.419392 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:33.419370 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:33.421259 22839 server_base.cc:1034] running on GCE node
I20260929 15:50:33.422377 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:33.424084 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:33.425433 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790697033425351 us; error 52 us; skew 500 ppm
I20260929 15:50:33.425877 22839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:33 dist-test-slave-7491 krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:33.430085 22839 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:33.432487 22839 webserver.cc:460] Webserver started at http://127.0.97.62:41993/ using document root <none> and password file <none>
I20260929 15:50:33.433499 22839 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:33.433660 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:33.434070 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:33.438426 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "9284b9bea93442ccbf2a2597c7e3ad4f"
format_stamp: "Formatted at 2026-09-29 15:50:33 on dist-test-slave-7491"
server_key: "f5c995d32fd5ab277b58681539b413b8"
server_key_iv: "c9f6d5ea28a311c773e810d89412930d"
server_key_version: "encryptionkey@0"
I20260929 15:50:33.439612 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "9284b9bea93442ccbf2a2597c7e3ad4f"
format_stamp: "Formatted at 2026-09-29 15:50:33 on dist-test-slave-7491"
server_key: "f5c995d32fd5ab277b58681539b413b8"
server_key_iv: "c9f6d5ea28a311c773e810d89412930d"
server_key_version: "encryptionkey@0"
I20260929 15:50:33.446388 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:50:33.450762 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:33.452857 22839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:50:33.453220 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "9284b9bea93442ccbf2a2597c7e3ad4f"
format_stamp: "Formatted at 2026-09-29 15:50:33 on dist-test-slave-7491"
server_key: "f5c995d32fd5ab277b58681539b413b8"
server_key_iv: "c9f6d5ea28a311c773e810d89412930d"
server_key_version: "encryptionkey@0"
I20260929 15:50:33.453596 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:33.484616 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:33.489845 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:33.490190 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:33.514242 22839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39637
I20260929 15:50:33.514293 22905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39637 every 8 connection(s)
I20260929 15:50:33.515910 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:33.517679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22839
I20260929 15:50:33.517930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:33.518664 388 external_mini_cluster.cc:1428] Setting key dfe3bff905ff810d5172423f139e3992
I20260929 15:50:33.521191 22906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:33.529632 22906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 15:50:33 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:50:33.560178 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.535076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39637 (local address 127.0.0.1:58484)
0929 15:50:33.537223 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.537292 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:33.539745 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:33.541319 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:33.541326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:33.542024 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:33.542407 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:33.542428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:33.545092 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:33.545097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:33.546538 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:33.546557 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:33.547659 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:33.547665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:33.547798 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:33.548775 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:33.548813 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:33.551870 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:33.555514 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:33.555527 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:33.555541 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:33.555967 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:33.556494 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:33.556499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:33.556503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:33.556665 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:33.557536 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:33.557546 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:33.557785 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 15:50:33.558046 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 15:50:33.558739 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1609,"mutex_wait_us":158,"spinlock_wait_cycles":128256,"thread_start_us":1089,"threads_started":1}
I20260929 15:50:33.560225 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.535076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.0.97.62:39637)
0929 15:50:33.536668 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:33.536689 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:33.537470 (+ 781us) server_negotiation.cc:407] Connection header received
0929 15:50:33.540492 (+ 3022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:33.540508 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:33.540940 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:33.541249 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:33.542683 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:33.544859 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:33.546746 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:33.547381 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:33.547545 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:33.552157 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:33.552219 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:33.552242 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:33.552283 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:33.555343 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:33.556111 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:33.556127 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:33.556130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:33.556367 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:33.556782 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:33.556789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:33.556792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:33.557382 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:33.557604 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:33.558180 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 15:50:33.558568 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":630,"threads_started":1}
I20260929 15:50:33.563524 22906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:50:33.564916 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.564430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45546)
0929 15:50:33.564646 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.564769 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 15:50:33.565264 22858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
I20260929 15:50:33.567072 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.566649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45550)
0929 15:50:33.566825 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.566913 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:50:33.567297 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 15:50:33.588763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41597
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39637,127.0.97.61:41597,127.0.97.60:35945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:50:33.616664 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } attempt: 1
I20260929 15:50:33.617844 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.617393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45560)
0929 15:50:33.617592 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.617702 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:50:33.619019 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.618608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45574)
0929 15:50:33.618768 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.618853 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 15:50:33.619283 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
I20260929 15:50:33.687619 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } attempt: 2
I20260929 15:50:33.689005 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.688503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45588)
0929 15:50:33.688752 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.688865 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 15:50:33.690376 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.689861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45602)
0929 15:50:33.690128 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.690213 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260929 15:50:33.690670 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
I20260929 15:50:33.781986 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } attempt: 3
I20260929 15:50:33.783298 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.782821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45616)
0929 15:50:33.783065 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.783158 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 15:50:33.784430 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.784022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45630)
0929 15:50:33.784197 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.784286 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 15:50:33.784658 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
I20260929 15:50:33.928020 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } attempt: 4
I20260929 15:50:33.929771 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.929056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45642)
0929 15:50:33.929455 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.929581 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260929 15:50:33.931411 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:33.930906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45652)
0929 15:50:33.931111 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:33.931223 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 15:50:33.931717 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41597: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41597: connect: Connection refused (error 111)
W20260929 15:50:33.990828 22913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:50:33.991186 22913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:33.991338 22913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:33.991477 22913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:34.000049 22913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:34.000231 22913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:34.000303 22913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:34.000355 22913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:34.000491 22913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:34.014864 22913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39637,127.0.97.61:41597,127.0.97.60:35945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:34.017081 22913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:34.019150 22913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:34.034186 22919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:34.034210 22918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:34.034143 22921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:34.038177 22913 server_base.cc:1034] running on GCE node
I20260929 15:50:34.039268 22913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:34.041172 22913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:34.042553 22913 hybrid_clock.cc:648] HybridClock initialized: now 1790697034042433 us; error 46 us; skew 500 ppm
I20260929 15:50:34.043279 22913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:34.047676 22913 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 15:50:34.049490 22913 webserver.cc:460] Webserver started at http://127.0.97.61:46471/ using document root <none> and password file <none>
I20260929 15:50:34.050297 22913 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:34.050398 22913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:34.050689 22913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:34.054057 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data/instance:
uuid: "59e3bfcde33b42bbbeacf0e35367536a"
format_stamp: "Formatted at 2026-09-29 15:50:34 on dist-test-slave-7491"
server_key: "ff4d3bda64f7d5f89453ac6ed9446399"
server_key_iv: "ac5aface30c83f9c36ecd69d8069607d"
server_key_version: "encryptionkey@0"
I20260929 15:50:34.055123 22913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal/instance:
uuid: "59e3bfcde33b42bbbeacf0e35367536a"
format_stamp: "Formatted at 2026-09-29 15:50:34 on dist-test-slave-7491"
server_key: "ff4d3bda64f7d5f89453ac6ed9446399"
server_key_iv: "ac5aface30c83f9c36ecd69d8069607d"
server_key_version: "encryptionkey@0"
I20260929 15:50:34.061345 22913 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:50:34.066140 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:34.068028 22913 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:50:34.068356 22913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal
uuid: "59e3bfcde33b42bbbeacf0e35367536a"
format_stamp: "Formatted at 2026-09-29 15:50:34 on dist-test-slave-7491"
server_key: "ff4d3bda64f7d5f89453ac6ed9446399"
server_key_iv: "ac5aface30c83f9c36ecd69d8069607d"
server_key_version: "encryptionkey@0"
I20260929 15:50:34.068615 22913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:34.097681 22913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:34.103073 22913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:34.103421 22913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:34.125777 22913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41597
I20260929 15:50:34.125787 22979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41597 every 8 connection(s)
I20260929 15:50:34.127319 22913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:50:34.131712 22980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:34.132210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22913
I20260929 15:50:34.132413 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-1/wal/instance
I20260929 15:50:34.132997 388 external_mini_cluster.cc:1428] Setting key d56711f04eddffd2be798644f36e49b3
I20260929 15:50:34.144759 22980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 15:50:34.176016 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.149482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39637 (local address 127.0.0.1:58488)
0929 15:50:34.151247 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.151293 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.153858 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.154807 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.154816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.155413 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.156402 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.156449 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.157805 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.157810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.159242 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.159261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.160219 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.160230 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.160502 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.161332 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.161359 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.164922 (+ 3563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.170264 (+ 5342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.170279 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.170308 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.171192 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.171789 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.171798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.171805 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.172108 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.172954 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.172964 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.173593 (+ 629us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.173920 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.174659 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"thread_start_us":1263,"threads_started":1}
I20260929 15:50:34.181615 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.148768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.0.97.62:39637)
0929 15:50:34.150872 (+ 2104us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.150881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.151442 (+ 561us) server_negotiation.cc:407] Connection header received
0929 15:50:34.154255 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.154261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.154366 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.154497 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.156838 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.157591 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.159441 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.159896 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.160044 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.165355 (+ 5311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.165375 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.165380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.165455 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.169958 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.171420 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.171430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.171438 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.171621 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.172329 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.172336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.172343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.172779 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.172959 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.180645 (+ 7686us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.181066 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1950,"thread_start_us":1830,"threads_started":1}
I20260929 15:50:34.184234 22980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:34.206586 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.185277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45656)
0929 15:50:34.185464 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.185488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.185621 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.186906 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.186914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.187522 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.187807 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.187823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.190203 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.190208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.191389 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.191422 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.193668 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.193674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.193942 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.194805 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.194826 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.197443 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.202658 (+ 5215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.202668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.202676 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.203354 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.204135 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.204141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.204145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.204537 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.205599 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.205608 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.205775 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.205904 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.206225 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 15:50:34.206759 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.185599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.0.97.61:41597)
0929 15:50:34.186388 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.186395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.186425 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:50:34.186475 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.186481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.186693 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.186823 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.188038 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.189913 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.191730 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.193234 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.193460 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.197858 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.197898 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.197925 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.198000 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.202257 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.203612 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.203629 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.203633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.203838 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.204718 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.204723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.204725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.205333 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.205563 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.206104 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.206372 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":478,"threads_started":1}
I20260929 15:50:34.209816 22980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 15:50:34.211810 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.211084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43740)
0929 15:50:34.211378 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.211584 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 15:50:34.212343 22930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.212469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35945
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39637,127.0.97.61:41597,127.0.97.60:35945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:50:34.214766 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.213908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43756)
0929 15:50:34.214239 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.214449 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260929 15:50:34.215093 22980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.231079 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } attempt: 5
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:34.248876 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.233503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.0.97.61:41597)
0929 15:50:34.233720 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.233725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.233746 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:50:34.233815 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.233820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.233923 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.234069 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.235166 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.235959 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.237938 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.238455 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.238628 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.242398 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.242404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.242407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.242446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.245837 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.246710 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.246715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.246718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.246861 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.247267 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.247277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.247279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.247614 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.247793 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.248269 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.248522 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:50:34.248880 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.232287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45658)
0929 15:50:34.232952 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.232990 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.233173 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.234116 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.234122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.234709 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.234959 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.234970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.236220 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.236227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.237597 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.237632 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.238655 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.238662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.238880 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.239662 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.239682 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.242103 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.246053 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.246059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.246064 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.246545 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.246982 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.246987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.246991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.247135 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.247743 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.247748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.247927 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.248114 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.248495 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260929 15:50:34.249953 22906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:50:34.251168 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.250704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43764)
0929 15:50:34.250920 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.251013 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 15:50:34.252389 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.251897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43774)
0929 15:50:34.252057 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.252147 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 15:50:34.252631 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.264472 22980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 1
I20260929 15:50:34.265864 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.265350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43786)
0929 15:50:34.265587 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.265698 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 15:50:34.266973 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.266575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43798)
0929 15:50:34.266766 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.266847 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 15:50:34.267228 22980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.304055 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 1
I20260929 15:50:34.305454 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.304927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43814)
0929 15:50:34.305162 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.305273 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 15:50:34.306532 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.306179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43820)
0929 15:50:34.306332 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.306416 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:50:34.306784 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.335516 22980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 2
I20260929 15:50:34.337199 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.336619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43832)
0929 15:50:34.336854 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.336960 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 15:50:34.338863 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.338397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43836)
0929 15:50:34.338617 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.338716 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260929 15:50:34.339251 22980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.375043 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 2
I20260929 15:50:34.376847 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.376101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43840)
0929 15:50:34.376558 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.376672 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260929 15:50:34.378088 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.377554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43844)
0929 15:50:34.377704 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.377954 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 15:50:34.378360 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.430491 22980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 3
I20260929 15:50:34.431823 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.431210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43852)
0929 15:50:34.431548 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.431652 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 15:50:34.432999 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.432536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43858)
0929 15:50:34.432767 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.432853 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260929 15:50:34.433234 22980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.484655 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 3
I20260929 15:50:34.486233 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.485626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43868)
0929 15:50:34.485886 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.486025 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 15:50:34.487815 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.487258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43870)
0929 15:50:34.487549 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.487652 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260929 15:50:34.488060 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
I20260929 15:50:34.576495 22980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 4
I20260929 15:50:34.577818 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.577202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43874)
0929 15:50:34.577450 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.577572 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 15:50:34.579102 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.578669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43888)
0929 15:50:34.578855 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.578950 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 15:50:34.579437 22980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
W20260929 15:50:34.627971 22988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:50:34.628441 22988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:34.628597 22988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:34.628774 22988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:34.637887 22988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:34.638019 22988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:34.638126 22988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:34.638217 22988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:34.638337 22988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:34.649356 22988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39637,127.0.97.61:41597,127.0.97.60:35945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:34.651101 22988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:34.652689 22988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:50:34.665345 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 4
I20260929 15:50:34.667109 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.666460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43898)
0929 15:50:34.666754 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.666892 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 15:50:34.668363 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.667910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43902)
0929 15:50:34.668070 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.668184 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 15:50:34.668772 22906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35945: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35945: connect: Connection refused (error 111)
W20260929 15:50:34.669169 22994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:34.669866 22993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:34.670312 22996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:34.670765 22988 server_base.cc:1034] running on GCE node
I20260929 15:50:34.671813 22988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:34.673714 22988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:34.675022 22988 hybrid_clock.cc:648] HybridClock initialized: now 1790697034674926 us; error 68 us; skew 500 ppm
I20260929 15:50:34.675463 22988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:34.679996 22988 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 15:50:34.681953 22988 webserver.cc:460] Webserver started at http://127.0.97.60:33345/ using document root <none> and password file <none>
I20260929 15:50:34.682835 22988 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:34.682988 22988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:34.683351 22988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:34.686841 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data/instance:
uuid: "c8b9094d0a02436ea86c985b4e0e892c"
format_stamp: "Formatted at 2026-09-29 15:50:34 on dist-test-slave-7491"
server_key: "c780a17645042cba5926026743ad9c1c"
server_key_iv: "ccf9b4d633f7fcfbdbe437337923af2d"
server_key_version: "encryptionkey@0"
I20260929 15:50:34.687834 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal/instance:
uuid: "c8b9094d0a02436ea86c985b4e0e892c"
format_stamp: "Formatted at 2026-09-29 15:50:34 on dist-test-slave-7491"
server_key: "c780a17645042cba5926026743ad9c1c"
server_key_iv: "ccf9b4d633f7fcfbdbe437337923af2d"
server_key_version: "encryptionkey@0"
I20260929 15:50:34.694375 22988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:50:34.699779 23003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:34.702244 22988 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:50:34.702644 22988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal
uuid: "c8b9094d0a02436ea86c985b4e0e892c"
format_stamp: "Formatted at 2026-09-29 15:50:34 on dist-test-slave-7491"
server_key: "c780a17645042cba5926026743ad9c1c"
server_key_iv: "ccf9b4d633f7fcfbdbe437337923af2d"
server_key_version: "encryptionkey@0"
I20260929 15:50:34.703109 22988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:34.719614 22988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:34.725487 22988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:34.725777 22988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:34.746419 22988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35945
I20260929 15:50:34.746436 23054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35945 every 8 connection(s)
I20260929 15:50:34.748029 22988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:50:34.752519 23055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:34.757536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 22988
I20260929 15:50:34.757702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790696645767654-388-0/minicluster-data/master-2/wal/instance
I20260929 15:50:34.758198 388 external_mini_cluster.cc:1428] Setting key edaa8b5c6f2e0690730c284d6987b636
I20260929 15:50:34.763562 23055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:50:34.787914 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.767429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.0.97.62:39637)
0929 15:50:34.768117 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.768125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.769005 (+ 880us) server_negotiation.cc:407] Connection header received
0929 15:50:34.769211 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.769217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.769303 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.769440 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.771224 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.772547 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.774786 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.775253 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.775386 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.780689 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.780695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.780698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.780768 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.784202 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.785207 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.785216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.785223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.785425 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.785989 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.785992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.785995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.786327 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.786467 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.787275 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.787561 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":315,"threads_started":1}
I20260929 15:50:34.789254 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.760840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.0.97.60:35945)
0929 15:50:34.761684 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.761702 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.761758 (+ 56us) server_negotiation.cc:407] Connection header received
0929 15:50:34.764115 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.764151 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.764806 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.765328 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.767636 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.769454 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.771550 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.773168 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.773330 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.779894 (+ 6564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.779908 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.779928 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.779980 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.783457 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.784545 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.784557 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.784562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.784771 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.785396 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.785402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.785405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.786507 (+ 1102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.786861 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.787181 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.787621 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":589,"thread_start_us":345,"threads_started":1}
I20260929 15:50:34.789249 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.767419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39637 (local address 127.0.0.1:58496)
0929 15:50:34.768809 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.768876 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.769069 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.769520 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.769526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.770577 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.770932 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.770948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.772899 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.772908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.774572 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.774594 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.775617 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.775631 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.775922 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.776941 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.776990 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.780396 (+ 3406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.784390 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.784402 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.784413 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.785037 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.785575 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.785580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.785583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.785790 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.786519 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.786527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.786784 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.787107 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.787952 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":354,"thread_start_us":923,"threads_started":1}
I20260929 15:50:34.792681 23055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:50:34.811007 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.793960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41597 (local address 127.0.0.1:45672)
0929 15:50:34.794341 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.794387 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.794553 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.795147 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.795151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.795688 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.795931 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.795948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.797537 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.797542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.798847 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.798868 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.799898 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.799903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.800073 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.800830 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.800851 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.803574 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.807887 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.807892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.807895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.808480 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.809098 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.809102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.809105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.809261 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.809937 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.809942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.810142 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.810314 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.810598 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260929 15:50:34.811050 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.794031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.0.97.61:41597)
0929 15:50:34.794794 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.794800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.794817 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:50:34.794876 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.794881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.794959 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.795067 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.796162 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.797339 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.799086 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.799665 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.799850 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.803873 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.803881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.803886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.803936 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.807641 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.808729 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.808738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.808744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.808921 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.809399 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.809404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.809407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.809787 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.809948 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.810439 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.810705 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":517,"threads_started":1}
I20260929 15:50:34.812780 23055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:50:34.813222 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.794178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.0.97.62:39637)
0929 15:50:34.794376 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.794383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.794753 (+ 370us) server_negotiation.cc:407] Connection header received
0929 15:50:34.794845 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.794857 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.794951 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.795064 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.796673 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.797890 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.799727 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.800350 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.800501 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.805415 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.805425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.805430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.805479 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.808986 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.809943 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.809948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.809951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.810140 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.810606 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.810611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.810614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.811000 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.811332 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.812360 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.812849 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:34.830600 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.813956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.0.97.60:35945)
0929 15:50:34.814111 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.814115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.814130 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:50:34.814169 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.814172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.814224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.814371 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.815427 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.816528 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.818068 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.818481 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.818655 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.822854 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.822859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.822862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.822892 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.826302 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.827025 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.827033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.827037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.827166 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.827789 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.827793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.827795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.828092 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.828335 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.829948 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.830209 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:50:34.830716 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.813655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43910)
0929 15:50:34.813907 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.813930 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.814052 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.814461 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.814465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.815006 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.815221 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.815234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.816732 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.816737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.817873 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.817891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.818699 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.818704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.819135 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.819849 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.819871 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.822636 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.826473 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.826481 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.826485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.826895 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.827404 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.827410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.827416 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.827633 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.829468 (+ 1835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.829476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.829648 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.829857 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.830373 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":177}
I20260929 15:50:34.833380 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.815249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.0.97.61:41597)
0929 15:50:34.815452 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.815456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.815472 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:50:34.815735 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.815742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.815841 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.815975 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.817498 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.818445 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.820265 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.820946 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.821146 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.826684 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.826690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.826693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.826733 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.830294 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.831246 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.831252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.831258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.831417 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.831814 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.831821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.831824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.832124 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.832471 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.832771 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.833032 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:50:34.838243 23055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: Bootstrap starting.
I20260929 15:50:34.839449 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:34.842587 23055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:34.843881 23055 log.cc:824] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:34.843470 22881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58504
I20260929 15:50:34.847236 23055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: No bootstrap required, opened a new log
I20260929 15:50:34.856139 23055 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:34.856950 23055 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b9094d0a02436ea86c985b4e0e892c, State: Initialized, Role: FOLLOWER
I20260929 15:50:34.857627 23055 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:34.859122 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:34.859740 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:34.861063 23055 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:50:34.872499 23074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:50:34.872730 23074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:50:34.878739 22980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 5
I20260929 15:50:34.869673 22988 master_runner.cc:186] Error getting master registration for 127.0.97.62:39637: OK, instance_id { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" instance_seqno: 1790697033506945 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:50:34.882151 22988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:50:34.897141 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.879502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43926)
0929 15:50:34.879820 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.879840 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.879975 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.880429 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.880433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.881048 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.881263 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.881276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.882596 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.882604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.884265 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.884286 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.885115 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.885121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.885350 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.886131 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.886154 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.888881 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.893829 (+ 4948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.893838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.893844 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.894465 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.894965 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.894970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.894974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.895149 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.895961 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.895966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.896138 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.896340 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.896705 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260929 15:50:34.897163 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.879698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.0.97.60:35945)
0929 15:50:34.879910 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.879916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.879938 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:50:34.880156 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.880193 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.880277 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.880396 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.881433 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.882429 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.884418 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.884937 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.885103 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.889139 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.889145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.889148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.889185 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.893603 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.894635 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.894642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.894646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.894827 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.895283 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.895289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.895293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.895823 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.895992 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.896492 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.896773 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:50:34.903855 22980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: Bootstrap starting.
I20260929 15:50:34.908329 22980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:34.909773 22980 log.cc:824] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:34.913093 22980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: No bootstrap required, opened a new log
I20260929 15:50:34.920840 22980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:34.921656 22980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e3bfcde33b42bbbeacf0e35367536a, State: Initialized, Role: FOLLOWER
I20260929 15:50:34.922325 22980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:34.923382 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:34.923795 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:34.924762 22980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:50:34.936662 23088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:50:34.936836 23088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:50:34.933462 22913 master_runner.cc:186] Error getting master registration for 127.0.97.62:39637: OK, instance_id { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" instance_seqno: 1790697033506945 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:50:34.946039 22906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } attempt: 5
I20260929 15:50:34.948961 22913 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 15:50:34 dist-test-slave-7491 krb5kdc[22826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:50:34.965385 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.947011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35945 (local address 127.0.0.1:43940)
0929 15:50:34.947501 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:34.947527 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:34.947701 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:34.948103 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:34.948107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:34.948598 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:34.948807 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.948817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.950283 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.950291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.951455 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:34.951482 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.952814 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.952820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:34.953035 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.953854 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:34.953879 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:34.957197 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:34.961893 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.961900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.961922 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.962589 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.963040 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:34.963086 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:34.963092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:34.963336 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:34.964124 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:34.964132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:34.964345 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 15:50:34.964522 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 15:50:34.964932 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260929 15:50:34.965806 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:34.947237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.0.97.60:35945)
0929 15:50:34.947477 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:34.947483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:34.947814 (+ 331us) server_negotiation.cc:407] Connection header received
0929 15:50:34.947886 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:34.947892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:34.947969 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:34.948066 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:34.948976 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.950146 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.951759 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:34.952631 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:34.952768 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:34.957473 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:34.957483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:34.957487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:34.957527 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:34.961663 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.962718 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.962725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.962731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.962913 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:34.963457 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:34.963480 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:34.963486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:34.963942 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:34.964101 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:34.964713 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 15:50:34.965156 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 15:50:34.971634 22906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f: Bootstrap starting.
I20260929 15:50:34.976136 22906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:34.977667 22906 log.cc:824] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f: Log is configured to *not* fsync() on all Append() calls
W20260929 15:50:34.978176 23009 tablet.cc:2369] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:50:34.981715 22906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f: No bootstrap required, opened a new log
I20260929 15:50:34.990240 22906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:34.990736 22906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9284b9bea93442ccbf2a2597c7e3ad4f, State: Initialized, Role: FOLLOWER
I20260929 15:50:34.991343 22906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:34.992815 23091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:34.993629 23091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: This master's current role is: FOLLOWER
W20260929 15:50:34.993818 22860 tablet.cc:2369] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:50:34.994222 22906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:34.994297 22881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 15:50:34.994505 22881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:34.998965 22881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:35.002300 22955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59e3bfcde33b42bbbeacf0e35367536a"
I20260929 15:50:35.003044 22955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:35.008491 22955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9284b9bea93442ccbf2a2597c7e3ad4f in term 1.
I20260929 15:50:35.009606 22858 leader_election.cc:302] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e3bfcde33b42bbbeacf0e35367536a, 9284b9bea93442ccbf2a2597c7e3ad4f; no voters:
I20260929 15:50:35.011886 23030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c8b9094d0a02436ea86c985b4e0e892c"
I20260929 15:50:35.012625 23030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:35.018846 23030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9284b9bea93442ccbf2a2597c7e3ad4f in term 1.
I20260929 15:50:35.020097 22881 leader_election.cc:288] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [CANDIDATE]: Term 1 election: Requested vote from peers 59e3bfcde33b42bbbeacf0e35367536a (127.0.97.61:41597), c8b9094d0a02436ea86c985b4e0e892c (127.0.97.60:35945)
I20260929 15:50:35.027740 23091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:35.038705 23091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [term 1 LEADER]: Becoming Leader. State: Replica: 9284b9bea93442ccbf2a2597c7e3ad4f, State: Running, Role: LEADER
I20260929 15:50:35.040076 23091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } }
I20260929 15:50:35.053071 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9284b9bea93442ccbf2a2597c7e3ad4f. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.053882 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:35.057754 23105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 15:50:35.106787 22934 tablet.cc:2369] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:50:35.119164 23105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:35.120529 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:35.068746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.0.97.60:35945)
0929 15:50:35.069084 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.069094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.071069 (+ 1975us) server_negotiation.cc:407] Connection header received
0929 15:50:35.071669 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.071679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.071820 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.071922 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.077497 (+ 5575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:35.078922 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:35.082868 (+ 3946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:35.083593 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:35.083717 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:35.104007 (+ 20290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:35.104018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:35.104024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:35.104093 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:35.108541 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:35.110694 (+ 2153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:35.110701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:35.110704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:35.110897 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:35.111403 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:35.111409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:35.111412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:35.111848 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:35.112189 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:35.119616 (+ 7427us) server_negotiation.cc:299] Negotiation successful
0929 15:50:35.119942 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 15:50:35.120718 23113 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:50:35.125743 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:35.070146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.0.97.62:39637)
0929 15:50:35.070379 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.070386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.080885 (+ 10499us) server_negotiation.cc:407] Connection header received
0929 15:50:35.081114 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.081121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.081253 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.081421 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.092516 (+ 11095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:35.093783 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:35.095960 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:35.096723 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:35.096852 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:35.112795 (+ 15943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:35.112803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:35.112807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:35.112861 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:35.118049 (+ 5188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:35.122667 (+ 4618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:35.122676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:35.122683 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:35.122904 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:35.123501 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:35.123507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:35.123511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:35.123951 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:35.124308 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:35.124944 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 15:50:35.125357 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:50:35.148340 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:35.073883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.0.97.61:41597)
0929 15:50:35.074089 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.074099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.081622 (+ 7523us) server_negotiation.cc:407] Connection header received
0929 15:50:35.082947 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.082954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.083055 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.083136 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.094707 (+ 11571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:35.095972 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:35.109383 (+ 13411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:35.110183 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:35.110309 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:35.139804 (+ 29495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:35.139813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:35.139818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:35.139878 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:35.144244 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:35.145418 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:35.145427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:35.145431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:35.145613 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:35.146121 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:35.146127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:35.146130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:35.146599 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:35.146893 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:35.147557 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 15:50:35.147860 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 15:50:35.157725 23030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [term 1 FOLLOWER]: Refusing update from remote peer 9284b9bea93442ccbf2a2597c7e3ad4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:50:35.157732 22955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [term 1 FOLLOWER]: Refusing update from remote peer 9284b9bea93442ccbf2a2597c7e3ad4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:50:35.159076 23098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:50:35.159883 23098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:50:35.192040 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9284b9bea93442ccbf2a2597c7e3ad4f. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.192566 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:35.202775 23114 mvcc.cc:205] Tried to move back new op lower bound from 7334695055992336384 to 7334695055550926848. Current Snapshot: MvccSnapshot[applied={T|T < 7334695055992336384}]
I20260929 15:50:35.213147 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9284b9bea93442ccbf2a2597c7e3ad4f. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.213519 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:35.222184 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.222512 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:35.222950 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.223208 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:35.223156 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.223439 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:35.226239 23105 catalog_manager.cc:1347] Generated new cluster ID: 6babe5d491b441149e8c83d4277dcb71
I20260929 15:50:35.226372 23105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:35.232599 23063 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9284b9bea93442ccbf2a2597c7e3ad4f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39637 } } peers { permanent_uuid: "59e3bfcde33b42bbbeacf0e35367536a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41597 } } peers { permanent_uuid: "c8b9094d0a02436ea86c985b4e0e892c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35945 } } }
I20260929 15:50:35.232915 23063 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:35.289857 23105 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:35.291977 23105 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:35.324370 23105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9284b9bea93442ccbf2a2597c7e3ad4f: Generated new TSK 0
I20260929 15:50:35.325675 23105 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:50:35.476194 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.454808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.0.97.60:35945)
0929 15:50:35.455056 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.455064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.472907 (+ 17843us) server_negotiation.cc:407] Connection header received
0929 15:50:35.474619 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.474625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.474742 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.474819 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.475870 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43966: BlockingRecv error: recv got EOF from 127.0.0.1:43966 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 15:50:35.477339 23059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.452492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.97.62:39637)
0929 15:50:35.452729 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.452735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.471291 (+ 18556us) server_negotiation.cc:407] Connection header received
0929 15:50:35.473451 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.473464 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.473568 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.473650 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.477091 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58526: BlockingRecv error: recv got EOF from 127.0.0.1:58526 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 15:50:35.480147 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.468068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.97.61:41597)
0929 15:50:35.468365 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.468372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.478013 (+ 9641us) server_negotiation.cc:407] Connection header received
0929 15:50:35.478686 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.478693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.478786 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.478875 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.479891 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45686: BlockingRecv error: recv got EOF from 127.0.0.1:45686 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260929 15:50:35.537134 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.510385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.97.61:41597)
0929 15:50:35.510631 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.510639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.534520 (+ 23881us) server_negotiation.cc:407] Connection header received
0929 15:50:35.535176 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.535184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.535347 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.535434 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.536837 (+ 1403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45692: BlockingRecv error: recv got EOF from 127.0.0.1:45692 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 15:50:35.550809 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.548477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.0.97.61:41597)
0929 15:50:35.548662 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.548669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.549223 (+ 554us) server_negotiation.cc:407] Connection header received
0929 15:50:35.549415 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.549421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.549518 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.549649 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.550567 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45700: BlockingRecv error: recv got EOF from 127.0.0.1:45700 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 15:50:35.558825 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.556371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.97.61:41597)
0929 15:50:35.556588 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.556594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.557153 (+ 559us) server_negotiation.cc:407] Connection header received
0929 15:50:35.557366 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.557372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.557469 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.557587 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.558553 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45708: BlockingRecv error: recv got EOF from 127.0.0.1:45708 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 15:50:35.573362 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.571166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.0.97.61:41597)
0929 15:50:35.571481 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.571488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.571699 (+ 211us) server_negotiation.cc:407] Connection header received
0929 15:50:35.571884 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.571891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.571994 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.572130 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.573091 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45714: BlockingRecv error: recv got EOF from 127.0.0.1:45714 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260929 15:50:35.588850 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.586586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.0.97.61:41597)
0929 15:50:35.586814 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.586820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.587127 (+ 307us) server_negotiation.cc:407] Connection header received
0929 15:50:35.587381 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.587386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.587504 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.587600 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.588589 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45730: BlockingRecv error: recv got EOF from 127.0.0.1:45730 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260929 15:50:35.612254 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.609732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.0.97.61:41597)
0929 15:50:35.610020 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.610029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.610760 (+ 731us) server_negotiation.cc:407] Connection header received
0929 15:50:35.610952 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.610959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.611061 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.611191 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.611941 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45742: BlockingRecv error: recv got EOF from 127.0.0.1:45742 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 15:50:35.650548 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.648408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.0.97.61:41597)
0929 15:50:35.648664 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.648671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.649043 (+ 372us) server_negotiation.cc:407] Connection header received
0929 15:50:35.649225 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.649231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.649347 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.649483 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.650288 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45752: BlockingRecv error: recv got EOF from 127.0.0.1:45752 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 15:50:35.721129 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.718526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.0.97.61:41597)
0929 15:50:35.718776 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.718783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.719073 (+ 290us) server_negotiation.cc:407] Connection header received
0929 15:50:35.719338 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.719345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.719458 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.719576 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.720885 (+ 1309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45766: BlockingRecv error: recv got EOF from 127.0.0.1:45766 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260929 15:50:35.855513 23061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:35.853406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.0.97.61:41597)
0929 15:50:35.853659 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:35.853668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:35.854064 (+ 396us) server_negotiation.cc:407] Connection header received
0929 15:50:35.854302 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:35.854307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:35.854385 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:35.854508 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:35.855314 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45768: BlockingRecv error: recv got EOF from 127.0.0.1:45768 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 15:50:35.875957 23074 catalog_manager.cc:1259] Loaded cluster ID: 6babe5d491b441149e8c83d4277dcb71
I20260929 15:50:35.876108 23074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: loading cluster ID for follower catalog manager: success
I20260929 15:50:35.879173 23074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: acquiring CA information for follower catalog manager: success
I20260929 15:50:35.881588 23074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8b9094d0a02436ea86c985b4e0e892c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:50:35.939764 23088 catalog_manager.cc:1259] Loaded cluster ID: 6babe5d491b441149e8c83d4277dcb71
I20260929 15:50:35.939888 23088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: loading cluster ID for follower catalog manager: success
I20260929 15:50:35.942814 23088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: acquiring CA information for follower catalog manager: success
I20260929 15:50:35.944876 23088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59e3bfcde33b42bbbeacf0e35367536a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:50:36.125896 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:36.118006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.0.97.61:41597)
0929 15:50:36.118214 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:36.118219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:36.118405 (+ 186us) server_negotiation.cc:407] Connection header received
0929 15:50:36.118781 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:36.118788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:36.119076 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:36.119249 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:36.119891 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.120803 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.122795 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.123263 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.123400 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:36.124086 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:36.124843 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:36.124984 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:36.125471 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 15:50:36.125569 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:50:36.143637 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:36.134046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.0.97.60:35945)
0929 15:50:36.136649 (+ 2603us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:36.136657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:36.136686 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:50:36.136962 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:36.136971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:36.137254 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:36.137670 (+ 416us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:36.138305 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.139170 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.140829 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.141276 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.141460 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:36.142038 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:36.142805 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:36.143012 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:36.143080 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 15:50:36.143194 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2449,"thread_start_us":2344,"threads_started":1}
I20260929 15:50:36.166599 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:36.156365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.0.97.60:35945)
0929 15:50:36.156611 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:36.156619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:36.158787 (+ 2168us) server_negotiation.cc:407] Connection header received
0929 15:50:36.159401 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:36.159408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:36.159570 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:36.159673 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:36.160812 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.162016 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.164596 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.165204 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.165307 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:36.165760 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:36.165887 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:36.165994 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:36.166167 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 15:50:36.166260 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:50:36.166805 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:36.156310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.0.97.62:39637)
0929 15:50:36.157214 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:36.157223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:36.159660 (+ 2437us) server_negotiation.cc:407] Connection header received
0929 15:50:36.159991 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:36.160000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:36.160282 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:36.160497 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:36.161247 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.162310 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.164232 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.164769 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.164928 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:36.165551 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:36.166026 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:36.166148 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:36.166390 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 15:50:36.166489 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":555,"threads_started":1}
I20260929 15:50:36.167670 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:36.156586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.0.97.61:41597)
0929 15:50:36.156899 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:36.156907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:36.159428 (+ 2521us) server_negotiation.cc:407] Connection header received
0929 15:50:36.159724 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:36.159732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:36.159858 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:36.160212 (+ 354us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:36.161041 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.161971 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.165457 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:36.166107 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:36.166228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:36.166848 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:36.166988 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:36.167068 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:36.167297 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 15:50:36.167386 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:50:36.172590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22839
I20260929 15:50:36.174100 22901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:36.296337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22839
W20260929 15:50:36.323028 22932 connection.cc:535] server connection from 127.0.0.1:45658 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:50:36.323406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22913
I20260929 15:50:36.324934 22975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:36.448998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22913
I20260929 15:50:36.472852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 22988
I20260929 15:50:36.474444 23050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:36.592571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 22988
2026-09-29T15:50:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5064 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:50:36 dist-test-slave-7491 krb5kdc[23200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:50:36 dist-test-slave-7491 krb5kdc[23200](info): set up 2 sockets
Sep 29 15:50:36 dist-test-slave-7491 krb5kdc[23200](info): commencing operation
krb5kdc: starting...
W20260929 15:50:37.781766 388 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/test-user.keytab.
Sep 29 15:50:37 dist-test-slave-7491 krb5kdc[23200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 15:50:38.085920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41023
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41023,127.0.97.61:40395,127.0.97.60:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:50:38.481992 23213 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:50:38.482368 23213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:38.482437 23213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:38.482527 23213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:38.490430 23213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:38.490550 23213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:38.490608 23213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:38.490653 23213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:38.490715 23213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:38.501708 23213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41023,127.0.97.61:40395,127.0.97.60:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:38.503144 23213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:38.504541 23213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:38.518391 23218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:38.518352 23221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:38.518425 23219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:38.519528 23213 server_base.cc:1034] running on GCE node
I20260929 15:50:38.520584 23213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:38.522219 23213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:38.523536 23213 hybrid_clock.cc:648] HybridClock initialized: now 1790697038523450 us; error 67 us; skew 500 ppm
I20260929 15:50:38.523947 23213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:38 dist-test-slave-7491 krb5kdc[23200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:38.527707 23213 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 15:50:38.529856 23213 webserver.cc:460] Webserver started at http://127.0.97.62:35647/ using document root <none> and password file <none>
I20260929 15:50:38.530997 23213 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:38.531112 23213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:38.531486 23213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:38.534957 23213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data/instance:
uuid: "1393044e0b10478eb2d1c18a0bc982eb"
format_stamp: "Formatted at 2026-09-29 15:50:38 on dist-test-slave-7491"
server_key: "a07cdfd5ea3bd0b158ad646af7782f55"
server_key_iv: "78d8137af96aff2bf197f980d3d4d9f3"
server_key_version: "encryptionkey@0"
I20260929 15:50:38.535916 23213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal/instance:
uuid: "1393044e0b10478eb2d1c18a0bc982eb"
format_stamp: "Formatted at 2026-09-29 15:50:38 on dist-test-slave-7491"
server_key: "a07cdfd5ea3bd0b158ad646af7782f55"
server_key_iv: "78d8137af96aff2bf197f980d3d4d9f3"
server_key_version: "encryptionkey@0"
I20260929 15:50:38.543310 23213 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 15:50:38.547734 23228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:38.549973 23213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:50:38.550359 23213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal
uuid: "1393044e0b10478eb2d1c18a0bc982eb"
format_stamp: "Formatted at 2026-09-29 15:50:38 on dist-test-slave-7491"
server_key: "a07cdfd5ea3bd0b158ad646af7782f55"
server_key_iv: "78d8137af96aff2bf197f980d3d4d9f3"
server_key_version: "encryptionkey@0"
I20260929 15:50:38.550751 23213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:38.574301 23213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:38.579576 23213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:38.579914 23213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:38.601898 23213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41023
I20260929 15:50:38.601922 23279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41023 every 8 connection(s)
I20260929 15:50:38.603382 23213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/data/info.pb
I20260929 15:50:38.607842 23280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:38.608460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 23213
I20260929 15:50:38.608654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-0/wal/instance
I20260929 15:50:38.609190 388 external_mini_cluster.cc:1428] Setting key 8a56f5ffc011fa9b72874e40dd52057f
I20260929 15:50:38.619661 23280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:38 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 15:50:38.647648 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.623845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41023 (local address 127.0.0.1:32840)
0929 15:50:38.625125 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.625161 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:38.627079 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:38.628089 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:38.628096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:38.628894 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:38.629260 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:38.629279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:38.631061 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:38.631067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:38.632365 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:38.632381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:38.634147 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:38.634168 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:38.634445 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:38.635497 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:38.635542 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:38.638668 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:38.643140 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:38.643152 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:38.643163 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:38.643645 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:38.644225 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:38.644234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:38.644238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:38.644415 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:38.645518 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:38.645524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:38.645749 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 15:50:38.645980 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 15:50:38.646434 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":847,"threads_started":1}
I20260929 15:50:38.647675 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.623845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.0.97.62:41023)
0929 15:50:38.624812 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:38.624835 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:38.625307 (+ 472us) server_negotiation.cc:407] Connection header received
0929 15:50:38.627551 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:38.627558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:38.627848 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:38.628071 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:38.629498 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:38.630856 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:38.632525 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:38.633509 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:38.633897 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:38.639020 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:38.639029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:38.639043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:38.639077 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:38.642910 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:38.643829 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:38.643843 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:38.643849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:38.644051 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:38.644545 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:38.644554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:38.644560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:38.645337 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:38.645547 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:38.646181 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 15:50:38.646658 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":701,"thread_start_us":452,"threads_started":1}
I20260929 15:50:38.651125 23280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:50:38.652539 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.652023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36622)
0929 15:50:38.652298 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.652411 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 15:50:38.652940 23231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
I20260929 15:50:38.655045 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.654468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36632)
0929 15:50:38.654677 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.654795 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 15:50:38.655550 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40395: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 15:50:38.680405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40395
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41023,127.0.97.61:40395,127.0.97.60:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:50:38.705271 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } attempt: 1
I20260929 15:50:38.706895 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.706367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36644)
0929 15:50:38.706601 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.706701 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 15:50:38.708032 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.707643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36652)
0929 15:50:38.707800 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.707888 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:50:38.708246 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40395: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
I20260929 15:50:38.776986 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } attempt: 2
I20260929 15:50:38.778757 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.778046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36666)
0929 15:50:38.778421 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.778558 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260929 15:50:38.780140 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.779620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36674)
0929 15:50:38.779831 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.779941 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:50:38.780431 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40395: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
I20260929 15:50:38.871817 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } attempt: 3
I20260929 15:50:38.873489 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.872853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36688)
0929 15:50:38.873156 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.873299 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 15:50:38.875108 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:38.874488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36704)
0929 15:50:38.874787 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:38.874898 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 15:50:38.875398 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40395: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
I20260929 15:50:39.018683 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } attempt: 4
I20260929 15:50:39.019995 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.019445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36706)
0929 15:50:39.019772 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.019857 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 15:50:39.021133 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.020766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36708)
0929 15:50:39.020920 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.021009 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:50:39.021469 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40395: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40395: connect: Connection refused (error 111)
W20260929 15:50:39.065423 23287 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:50:39.065804 23287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:39.065927 23287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:39.066072 23287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:39.073794 23287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:39.073912 23287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:39.073972 23287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:39.074083 23287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:39.074190 23287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:39.084739 23287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41023,127.0.97.61:40395,127.0.97.60:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:39.087034 23287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:39.089066 23287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:39.105784 23293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:39.106359 23292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:39.105791 23295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:39.107518 23287 server_base.cc:1034] running on GCE node
I20260929 15:50:39.108898 23287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:39.111101 23287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:39.114629 23287 hybrid_clock.cc:648] HybridClock initialized: now 1790697039114380 us; error 168 us; skew 500 ppm
I20260929 15:50:39.115250 23287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:39.119940 23287 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 15:50:39.122341 23287 webserver.cc:460] Webserver started at http://127.0.97.61:33999/ using document root <none> and password file <none>
I20260929 15:50:39.123530 23287 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:39.123692 23287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:39.124114 23287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:39.127735 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data/instance:
uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4"
format_stamp: "Formatted at 2026-09-29 15:50:39 on dist-test-slave-7491"
server_key: "aec93e4517ec8cd02a5fca7eaa50c3e4"
server_key_iv: "488043fed3eb3d0b075f29747b508ba3"
server_key_version: "encryptionkey@0"
I20260929 15:50:39.128784 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal/instance:
uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4"
format_stamp: "Formatted at 2026-09-29 15:50:39 on dist-test-slave-7491"
server_key: "aec93e4517ec8cd02a5fca7eaa50c3e4"
server_key_iv: "488043fed3eb3d0b075f29747b508ba3"
server_key_version: "encryptionkey@0"
I20260929 15:50:39.135193 23287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 15:50:39.139430 23302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:39.141383 23287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:50:39.141631 23287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal
uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4"
format_stamp: "Formatted at 2026-09-29 15:50:39 on dist-test-slave-7491"
server_key: "aec93e4517ec8cd02a5fca7eaa50c3e4"
server_key_iv: "488043fed3eb3d0b075f29747b508ba3"
server_key_version: "encryptionkey@0"
I20260929 15:50:39.141927 23287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:39.164630 23287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:39.169668 23287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:39.169976 23287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:39.194245 23353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40395 every 8 connection(s)
I20260929 15:50:39.194288 23287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40395
I20260929 15:50:39.196367 23287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/data/info.pb
I20260929 15:50:39.201823 23354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:39.202207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 23287
I20260929 15:50:39.202405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-1/wal/instance
I20260929 15:50:39.202890 388 external_mini_cluster.cc:1428] Setting key 84e3146f3dc6a6fa0075e054807ae9ce
I20260929 15:50:39.209633 23354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 15:50:39.238597 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.212935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.97.62:41023)
0929 15:50:39.213684 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.213696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.214627 (+ 931us) server_negotiation.cc:407] Connection header received
0929 15:50:39.216913 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.216923 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.217233 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.217403 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.219308 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.221008 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.223242 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.224042 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.224270 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.229135 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.229141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.229144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.229201 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.232876 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.234206 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.234212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.234214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.234428 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.235159 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.235164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.235166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.235902 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.236070 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.237632 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.237959 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":341,"threads_started":1}
I20260929 15:50:39.240134 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.213482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41023 (local address 127.0.0.1:32850)
0929 15:50:39.214398 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.214449 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.216534 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.217711 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.217721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.218344 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.219005 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.219026 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.221320 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.221327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.222959 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.222985 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.224514 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.224521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.224816 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.225714 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.225740 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.228742 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.233239 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.233250 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.233268 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.233979 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.234718 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.234722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.234725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.234969 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.236223 (+ 1254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.236252 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.236910 (+ 658us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.237487 (+ 577us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.238181 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":395,"threads_started":1}
I20260929 15:50:39.242833 23354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:39.262964 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.243849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36712)
0929 15:50:39.244234 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.244258 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.244409 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.245263 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.245271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.245882 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.246104 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.246118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.247832 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.247840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.249499 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.249521 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.250493 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.250502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.250720 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.251604 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.251632 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.254532 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.259603 (+ 5071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.259609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.259615 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.260099 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.260672 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.260681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.260685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.260850 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.261869 (+ 1019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.261874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.262024 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.262182 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.262538 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260929 15:50:39.263077 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.243920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.0.97.61:40395)
0929 15:50:39.244769 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.244776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.244805 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:50:39.244855 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.244861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.245069 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.245211 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.246339 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.247644 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.249699 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.250312 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.250424 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.254799 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.254817 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.254833 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.254866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.259352 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.260348 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.260358 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.260360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.260543 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.261126 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.261136 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.261141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.261668 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.261963 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.262357 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.262670 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":668,"thread_start_us":513,"threads_started":1}
I20260929 15:50:39.266033 23354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:50:39.267262 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.266753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51354)
0929 15:50:39.266954 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.267098 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:50:39.267619 23306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.269631 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.269188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51364)
0929 15:50:39.269405 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.269506 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 15:50:39.269913 23354 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 15:50:39.272918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQOquZz/build/asan/bin/kudu
/tmp/dist-test-taskQOquZz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35439
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41023,127.0.97.61:40395,127.0.97.60:35439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:50:39.319329 23354 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 1
I20260929 15:50:39.320746 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } attempt: 5
I20260929 15:50:39.320932 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.320407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51372)
0929 15:50:39.320607 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.320725 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 15:50:39.322139 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.321760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51382)
0929 15:50:39.321876 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.321995 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 15:50:39.322396 23354 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:39.338852 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.321417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36728)
0929 15:50:39.321678 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.321697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.321856 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.322331 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.322336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.322872 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.323115 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.323128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.324716 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.324759 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.325746 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.325769 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.326815 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.326820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.326974 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.327711 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.327731 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.330158 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.335417 (+ 5259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.335423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.335426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.335928 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.336776 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.336782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.336785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.336996 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.337806 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.337810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.337977 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.338175 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.338529 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 15:50:39.338970 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.321517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.0.97.61:40395)
0929 15:50:39.321710 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.321717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.321765 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:50:39.322044 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.322049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.322134 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.323291 (+ 1157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.323439 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.324521 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.325952 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.326589 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.326727 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.330449 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.330456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.330461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.330514 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.335156 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.336239 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.336248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.336252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.336514 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.337145 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.337151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.337154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.337634 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.337804 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.338327 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.338645 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 15:50:39.340739 23280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
I20260929 15:50:39.342360 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.341815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51388)
0929 15:50:39.342044 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.342146 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 15:50:39.343451 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.343099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51394)
0929 15:50:39.343264 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.343334 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 15:50:39.343736 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.390865 23354 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 2
I20260929 15:50:39.392748 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.392054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51404)
0929 15:50:39.392415 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.392555 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260929 15:50:39.394059 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.393668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51412)
0929 15:50:39.393831 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.393925 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 15:50:39.394434 23354 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.394999 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 1
I20260929 15:50:39.396886 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.396105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51422)
0929 15:50:39.396458 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.396586 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 15:50:39.398078 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.397710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51436)
0929 15:50:39.397868 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.397963 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 15:50:39.398396 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.466706 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 2
I20260929 15:50:39.467995 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.467470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51442)
0929 15:50:39.467728 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.467826 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 15:50:39.469084 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.468747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51454)
0929 15:50:39.468899 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.468977 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 15:50:39.469393 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.485819 23354 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 3
I20260929 15:50:39.487282 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.486640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51464)
0929 15:50:39.486952 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.487083 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 15:50:39.488641 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.488065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51468)
0929 15:50:39.488357 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.488470 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260929 15:50:39.488950 23354 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.575711 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 3
I20260929 15:50:39.577169 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.576579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51472)
0929 15:50:39.576881 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.576983 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 15:50:39.578406 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.577945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51484)
0929 15:50:39.578192 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.578274 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 15:50:39.578692 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
I20260929 15:50:39.632267 23354 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 4
I20260929 15:50:39.633872 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.633228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51488)
0929 15:50:39.633518 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.633643 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 15:50:39.635324 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.634820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51500)
0929 15:50:39.635035 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.635146 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 15:50:39.635694 23354 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
W20260929 15:50:39.702438 23362 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 15:50:39.702831 23362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:50:39.703022 23362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:50:39.703217 23362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:50:39.712773 23362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:50:39.712909 23362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:50:39.712971 23362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:50:39.713016 23362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:50:39.713079 23362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:50:39.725567 23362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41023,127.0.97.61:40395,127.0.97.60:35439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:50:39.727022 23362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:50:39.728564 23362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:50:39.744738 23368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:39.744783 23370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:50:39.745425 23367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:50:39.745769 23362 server_base.cc:1034] running on GCE node
I20260929 15:50:39.746774 23362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:50:39.748596 23362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:50:39.749914 23362 hybrid_clock.cc:648] HybridClock initialized: now 1790697039749845 us; error 47 us; skew 500 ppm
I20260929 15:50:39.750339 23362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:50:39.755941 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 4
I20260929 15:50:39.757037 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.756595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51512)
0929 15:50:39.756802 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.756894 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 15:50:39.757977 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.757655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51524)
0929 15:50:39.757798 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.757872 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:50:39.758196 23280 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35439: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35439: connect: Connection refused (error 111)
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:50:39.759982 23362 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 15:50:39.761999 23362 webserver.cc:460] Webserver started at http://127.0.97.60:45561/ using document root <none> and password file <none>
I20260929 15:50:39.762825 23362 fs_manager.cc:359] Metadata directory not provided
I20260929 15:50:39.762982 23362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:50:39.763326 23362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:50:39.766641 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data/instance:
uuid: "aaaf2859b15e489fb97d685712accda0"
format_stamp: "Formatted at 2026-09-29 15:50:39 on dist-test-slave-7491"
server_key: "b6d80a0ef0a99658e4d71afe0941c912"
server_key_iv: "ebbb6132c0fbd2b04849006b92439957"
server_key_version: "encryptionkey@0"
I20260929 15:50:39.767582 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal/instance:
uuid: "aaaf2859b15e489fb97d685712accda0"
format_stamp: "Formatted at 2026-09-29 15:50:39 on dist-test-slave-7491"
server_key: "b6d80a0ef0a99658e4d71afe0941c912"
server_key_iv: "ebbb6132c0fbd2b04849006b92439957"
server_key_version: "encryptionkey@0"
I20260929 15:50:39.774950 23362 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260929 15:50:39.781188 23377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:50:39.783615 23362 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:50:39.783881 23362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal
uuid: "aaaf2859b15e489fb97d685712accda0"
format_stamp: "Formatted at 2026-09-29 15:50:39 on dist-test-slave-7491"
server_key: "b6d80a0ef0a99658e4d71afe0941c912"
server_key_iv: "ebbb6132c0fbd2b04849006b92439957"
server_key_version: "encryptionkey@0"
I20260929 15:50:39.784221 23362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:50:39.804201 23362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:50:39.811491 23362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:50:39.811864 23362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:50:39.839720 23428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35439 every 8 connection(s)
I20260929 15:50:39.839860 23362 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35439
I20260929 15:50:39.842443 23362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/data/info.pb
I20260929 15:50:39.848029 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:50:39.848454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQOquZz/build/asan/bin/kudu as pid 23362
I20260929 15:50:39.848649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQOquZz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790696645767654-388-0/minicluster-data/master-2/wal/instance
I20260929 15:50:39.849184 388 external_mini_cluster.cc:1428] Setting key 9cf22024da83bc72cefd30d4236be338
I20260929 15:50:39.862187 23429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 15:50:39.885769 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.851640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.0.97.60:35439)
0929 15:50:39.853032 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.853078 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.853162 (+ 84us) server_negotiation.cc:407] Connection header received
0929 15:50:39.857350 (+ 4188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.857404 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.858573 (+ 1169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.859253 (+ 680us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.861340 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.864097 (+ 2757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.868063 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.869416 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.869636 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.875202 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.875219 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.875239 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.875288 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.879337 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.880361 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.880377 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.880394 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.880682 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.881487 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.881495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.881499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.882464 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.882848 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.883539 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.884076 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":691,"threads_started":1}
I20260929 15:50:39.888263 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.866629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41023 (local address 127.0.0.1:32866)
0929 15:50:39.867601 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.867658 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.867889 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.868674 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.868684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.869850 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.870247 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.870264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.872271 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.872277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.874045 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.874068 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.875102 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.875109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.875284 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.876071 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.876090 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.879297 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.884645 (+ 5348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.884655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.884663 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.885297 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.885917 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.885921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.885925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.886119 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.886867 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.886877 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.887120 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.887351 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.887802 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":496,"threads_started":1}
I20260929 15:50:39.889134 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.866642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.0.97.62:41023)
0929 15:50:39.867286 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.867291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.867787 (+ 496us) server_negotiation.cc:407] Connection header received
0929 15:50:39.868024 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.868032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.868151 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.868336 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.870475 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.871727 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.874336 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.874788 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.874911 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.879590 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.879596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.879599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.879647 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.884453 (+ 4806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.885452 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.885459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.885462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.885669 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.886277 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.886283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.886286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.886680 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.886861 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.888494 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.888809 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":374,"threads_started":1}
I20260929 15:50:39.890802 23429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:39.909268 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.889706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.0.97.62:41023)
0929 15:50:39.889889 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.889894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.889984 (+ 90us) server_negotiation.cc:407] Connection header received
0929 15:50:39.890192 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.890197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.890300 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.890411 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.891841 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.892932 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.894482 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.895211 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.895587 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.902073 (+ 6486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.902081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.902084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.902132 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.905999 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.906878 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.906884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.906888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.907031 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.907512 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.907516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.907518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.907974 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.908338 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.908700 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.908955 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:50:39.910122 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.891715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40395 (local address 127.0.0.1:36736)
0929 15:50:39.892009 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.892040 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.892274 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.893113 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.893119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.893566 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.893736 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.893746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.895555 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.895564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.896916 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.896935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.898280 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.898287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.898547 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.899387 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.899408 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.902562 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.906895 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.906900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.906902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.907589 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.908137 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.908142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.908146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.908384 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.909132 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.909139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.909295 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.909462 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.909778 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 15:50:39.910435 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.891890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.97.61:40395)
0929 15:50:39.892623 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.892630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.892655 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:50:39.892721 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.892727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.892808 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.892941 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.893955 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.895197 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.897197 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.897969 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.898158 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.902841 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.902847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.902851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.902889 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.906688 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.907765 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.907771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.907791 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.907969 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.908513 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.908517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.908519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.908892 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.909051 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.909762 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.910016 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":410,"threads_started":1}
I20260929 15:50:39.911516 23429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 15:50:39.928608 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.912473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.0.97.60:35439)
0929 15:50:39.912696 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.912704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.912727 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:50:39.912783 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.912787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.912864 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.912971 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.913658 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.914604 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.916726 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.917320 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.917420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.921935 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.921941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.921945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.921986 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.925448 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.926413 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.926418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.926420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.926538 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.926986 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.926990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.926999 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.927317 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.927515 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.927873 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.928160 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":84}
I20260929 15:50:39.928675 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.912331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51542)
0929 15:50:39.912617 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.912631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.912711 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.913004 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.913007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.913379 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.913513 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.913521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.915118 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.915130 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.916545 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.916560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.917482 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.917489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.917833 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.918716 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.918736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.921608 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.925670 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.925678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.925683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.926238 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.926672 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.926677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.926680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.926854 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.927484 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.927489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.927626 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.927786 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.928334 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":176}
I20260929 15:50:39.930467 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.911313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.0.97.61:40395)
0929 15:50:39.911461 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.911467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.911485 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:50:39.911546 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.911551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.911626 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.911729 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.913960 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.915283 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.917165 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.917974 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.918151 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.923702 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.923708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.923711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.923753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.927211 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.927998 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.928005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.928010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.928145 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.928606 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.928611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.928615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.929194 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.929511 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.929796 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.930092 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 15:50:39.934953 23354 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 5
I20260929 15:50:39.935246 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: Bootstrap starting.
I20260929 15:50:39.938189 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:50:39.941686 23429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:39.943648 23429 log.cc:824] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:39.941169 23255 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1393044e0b10478eb2d1c18a0bc982eb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32876
Sep 29 15:50:39 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:50:39.948004 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: No bootstrap required, opened a new log
I20260929 15:50:39.956406 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.935651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51550)
0929 15:50:39.935944 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:39.935972 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:39.936136 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:39.936783 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:39.936788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:39.937335 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:39.937613 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.937628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.939311 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.939320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.941022 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:39.941073 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.942371 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.942379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:39.942642 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.943484 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:39.943511 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:39.946872 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:39.951999 (+ 5127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.952006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.952012 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.952611 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.953221 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:39.953227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:39.953230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:39.953435 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:39.954947 (+ 1512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:39.954953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:39.955192 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 15:50:39.955475 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 15:50:39.955910 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 15:50:39.956740 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:39.936076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.0.97.60:35439)
0929 15:50:39.936349 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:39.936355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:39.936427 (+ 72us) server_negotiation.cc:407] Connection header received
0929 15:50:39.936493 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:39.936498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:39.936591 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:39.937781 (+ 1190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:39.937938 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.939087 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.941285 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:39.942137 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:39.942328 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:39.947342 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:39.947348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:39.947352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:39.947414 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:39.951752 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.952809 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.952816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.952820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.953055 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:39.953587 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:39.953592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:39.953595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:39.954626 (+ 1031us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:39.955035 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:39.955699 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 15:50:39.956060 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:50:39.958724 23429 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:39.959098 23429 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaf2859b15e489fb97d685712accda0, State: Initialized, Role: FOLLOWER
I20260929 15:50:39.959628 23429 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:39.960973 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:39.961745 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:39.961835 23429 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:39.963186 23354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: Bootstrap starting.
I20260929 15:50:39.968083 23354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:39.969830 23354 log.cc:824] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: Log is configured to *not* fsync() on all Append() calls
W20260929 15:50:39.973385 23448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:50:39.973538 23448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:50:39.978222 23354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: No bootstrap required, opened a new log
I20260929 15:50:39.971204 23362 master_runner.cc:186] Error getting master registration for 127.0.97.62:41023: OK, instance_id { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" instance_seqno: 1790697038595539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:50:39.984696 23362 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:50:39.990971 23354 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:39.991447 23354 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6d87cd608642e58f1ecf0e3dad2fc4, State: Initialized, Role: FOLLOWER
I20260929 15:50:39.992060 23354 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:39.993244 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:39.993688 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:39.994822 23354 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:50:40.020270 23462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:50:40.020612 23462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:50:40.006978 23287 master_runner.cc:186] Error getting master registration for 127.0.97.62:41023: OK, instance_id { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" instance_seqno: 1790697038595539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 15:50:40.024278 23287 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 15:50:40.035465 23280 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } attempt: 5
Sep 29 15:50:40 dist-test-slave-7491 krb5kdc[23200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 15:50:40.056072 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:40.036524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35439 (local address 127.0.0.1:51558)
0929 15:50:40.036901 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 15:50:40.036924 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:50:40.037095 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:50:40.037521 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:50:40.037526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:50:40.038250 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:50:40.038556 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:40.038574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.039895 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.039903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:40.042176 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:50:40.042208 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.043219 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.043226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:50:40.043444 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:40.044265 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:50:40.044290 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:50:40.047329 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:50:40.051278 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:40.051286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:40.051291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:40.051890 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:40.052726 (+ 836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:50:40.052734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:50:40.052739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:50:40.053173 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:50:40.054389 (+ 1216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:50:40.054397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:50:40.054793 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 15:50:40.055141 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 15:50:40.055562 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260929 15:50:40.057282 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:40.036652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.60:35439)
0929 15:50:40.036940 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.036948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.036989 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:50:40.037269 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.037278 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.037360 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.037491 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.038845 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.039730 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.042442 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.043016 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.043194 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:40.047618 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:40.047624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:40.047628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:40.047762 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:40.051067 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.052031 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.052045 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.052049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.052421 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.053354 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.053360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.053364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.053913 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:40.054276 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:40.056523 (+ 2247us) server_negotiation.cc:299] Negotiation successful
0929 15:50:40.056858 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:50:40.063851 23280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb: Bootstrap starting.
W20260929 15:50:40.064725 23383 tablet.cc:2369] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:50:40.068890 23280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb: Neither blocks nor log segments found. Creating new log.
I20260929 15:50:40.070399 23280 log.cc:824] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb: Log is configured to *not* fsync() on all Append() calls
I20260929 15:50:40.076146 23280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb: No bootstrap required, opened a new log
I20260929 15:50:40.086591 23280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:40.087081 23280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1393044e0b10478eb2d1c18a0bc982eb, State: Initialized, Role: FOLLOWER
I20260929 15:50:40.087684 23280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:40.089519 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.090294 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:40.090826 23280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:50:40.090895 23255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 15:50:40.091097 23255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:40.096865 23255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:40.100513 23329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1393044e0b10478eb2d1c18a0bc982eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4"
I20260929 15:50:40.100425 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1393044e0b10478eb2d1c18a0bc982eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aaaf2859b15e489fb97d685712accda0"
I20260929 15:50:40.101051 23329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:40.101109 23404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:50:40.102123 23255 leader_election.cc:288] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [CANDIDATE]: Term 1 election: Requested vote from peers ce6d87cd608642e58f1ecf0e3dad2fc4 (127.0.97.61:40395), aaaf2859b15e489fb97d685712accda0 (127.0.97.60:35439)
I20260929 15:50:40.105230 23329 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1393044e0b10478eb2d1c18a0bc982eb in term 1.
I20260929 15:50:40.105228 23404 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1393044e0b10478eb2d1c18a0bc982eb in term 1.
I20260929 15:50:40.106041 23231 leader_election.cc:302] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1393044e0b10478eb2d1c18a0bc982eb, ce6d87cd608642e58f1ecf0e3dad2fc4; no voters:
I20260929 15:50:40.111279 23465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:50:40.121155 23465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [term 1 LEADER]: Becoming Leader. State: Replica: 1393044e0b10478eb2d1c18a0bc982eb, State: Running, Role: LEADER
I20260929 15:50:40.122480 23465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } }
I20260929 15:50:40.135439 23468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1393044e0b10478eb2d1c18a0bc982eb. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.136034 23468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:40.142880 23475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 15:50:40.173465 23308 tablet.cc:2369] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:50:40.176069 23475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:50:40.206813 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:40.164382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.0.97.60:35439)
0929 15:50:40.164777 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.164787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.173658 (+ 8871us) server_negotiation.cc:407] Connection header received
0929 15:50:40.174446 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.174455 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.174587 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.174672 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.183487 (+ 8815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.184718 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.186755 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.187487 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.187602 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:40.193519 (+ 5917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:40.193527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:40.193535 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:40.193594 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:40.198908 (+ 5314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.202650 (+ 3742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.202659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.202663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.203031 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.204194 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.204205 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.204208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.204696 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:40.205019 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:40.206070 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0929 15:50:40.206397 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260929 15:50:40.222302 23487 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:50:40.238300 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:40.154517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.97.61:40395)
0929 15:50:40.154791 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.154798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.165327 (+ 10529us) server_negotiation.cc:407] Connection header received
0929 15:50:40.172418 (+ 7091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.172430 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.172582 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.172693 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.184335 (+ 11642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.185488 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.188254 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.189103 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.189219 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:40.193263 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:40.193271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:40.193276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:40.193358 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:40.199266 (+ 5908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.232993 (+ 33727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.233000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.233004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.233234 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.234608 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.234616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.234619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.235095 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:40.235428 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:40.237544 (+ 2116us) server_negotiation.cc:299] Negotiation successful
0929 15:50:40.237873 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 15:50:40.241415 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:40.156730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.0.97.62:41023)
0929 15:50:40.157135 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.157155 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.169073 (+ 11918us) server_negotiation.cc:407] Connection header received
0929 15:50:40.180655 (+ 11582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.180665 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.180803 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.180900 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.187714 (+ 6814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.201359 (+ 13645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.205336 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:40.206198 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:40.206311 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:40.212986 (+ 6675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:50:40.212997 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:50:40.213003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:50:40.213078 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:50:40.221403 (+ 8325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.233896 (+ 12493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.233902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.233905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.234078 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:50:40.235982 (+ 1904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:50:40.235987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:50:40.235990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:50:40.236460 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:50:40.236885 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:40.240741 (+ 3856us) server_negotiation.cc:299] Negotiation successful
0929 15:50:40.241028 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:50:40.241415 23329 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [term 1 FOLLOWER]: Refusing update from remote peer 1393044e0b10478eb2d1c18a0bc982eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:50:40.244683 23465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:50:40.246347 23404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [term 1 FOLLOWER]: Refusing update from remote peer 1393044e0b10478eb2d1c18a0bc982eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:50:40.248356 23465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:50:40.287066 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1393044e0b10478eb2d1c18a0bc982eb. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.287441 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:40.289671 23488 mvcc.cc:205] Tried to move back new op lower bound from 7334695076811341824 to 7334695076374097920. Current Snapshot: MvccSnapshot[applied={T|T < 7334695076811341824}]
I20260929 15:50:40.290676 23489 mvcc.cc:205] Tried to move back new op lower bound from 7334695076811341824 to 7334695076374097920. Current Snapshot: MvccSnapshot[applied={T|T < 7334695076811341824}]
I20260929 15:50:40.295938 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1393044e0b10478eb2d1c18a0bc982eb. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.296427 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:40.301503 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.301833 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:40.304822 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.305114 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:50:40.312033 23475 catalog_manager.cc:1347] Generated new cluster ID: 5812ccc4c9ac47eb8cc99ee55cd0e77c
I20260929 15:50:40.312209 23475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:50:40.314174 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.314507 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:40.314754 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1393044e0b10478eb2d1c18a0bc982eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1393044e0b10478eb2d1c18a0bc982eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41023 } } peers { permanent_uuid: "ce6d87cd608642e58f1ecf0e3dad2fc4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40395 } } peers { permanent_uuid: "aaaf2859b15e489fb97d685712accda0" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35439 } } }
I20260929 15:50:40.314937 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb [sys.catalog]: This master's current role is: LEADER
I20260929 15:50:40.340904 23475 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:50:40.342901 23475 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:50:40.362568 23475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1393044e0b10478eb2d1c18a0bc982eb: Generated new TSK 0
I20260929 15:50:40.363950 23475 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:50:40.507980 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.494070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.0.97.61:40395)
0929 15:50:40.494313 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.494320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.503557 (+ 9237us) server_negotiation.cc:407] Connection header received
0929 15:50:40.506124 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.506132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.506226 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.506314 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.507532 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36760: BlockingRecv error: recv got EOF from 127.0.0.1:36760 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 15:50:40.510576 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.498123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.0.97.62:41023)
0929 15:50:40.498402 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.498414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.505049 (+ 6635us) server_negotiation.cc:407] Connection header received
0929 15:50:40.508581 (+ 3532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.508587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.508700 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.509009 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.510306 (+ 1297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32892: BlockingRecv error: recv got EOF from 127.0.0.1:32892 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 15:50:40.513897 23432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.504284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.0.97.60:35439)
0929 15:50:40.504463 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.504470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.511893 (+ 7423us) server_negotiation.cc:407] Connection header received
0929 15:50:40.512297 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.512304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.512400 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.512485 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.513565 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51572: BlockingRecv error: recv got EOF from 127.0.0.1:51572 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 15:50:40.561525 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.550841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.0.97.61:40395)
0929 15:50:40.551077 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.551086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.559107 (+ 8021us) server_negotiation.cc:407] Connection header received
0929 15:50:40.559722 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.559744 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.559887 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.559975 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.561234 (+ 1259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36774: BlockingRecv error: recv got EOF from 127.0.0.1:36774 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 15:50:40.619792 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.594670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.0.97.61:40395)
0929 15:50:40.594925 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.594931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.617545 (+ 22614us) server_negotiation.cc:407] Connection header received
0929 15:50:40.618136 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.618143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.618256 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.618344 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.619493 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36784: BlockingRecv error: recv got EOF from 127.0.0.1:36784 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260929 15:50:40.654768 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.648489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.0.97.61:40395)
0929 15:50:40.648750 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.648757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.653071 (+ 4314us) server_negotiation.cc:407] Connection header received
0929 15:50:40.653301 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.653307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.653421 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.653566 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.654481 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36790: BlockingRecv error: recv got EOF from 127.0.0.1:36790 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 15:50:40.665786 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.663262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.0.97.61:40395)
0929 15:50:40.663544 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.663552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.664116 (+ 564us) server_negotiation.cc:407] Connection header received
0929 15:50:40.664408 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.664414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.664511 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.664627 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.665516 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36792: BlockingRecv error: recv got EOF from 127.0.0.1:36792 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260929 15:50:40.680225 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.677821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.0.97.61:40395)
0929 15:50:40.678011 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.678017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.678576 (+ 559us) server_negotiation.cc:407] Connection header received
0929 15:50:40.678791 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.678797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.678891 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.679002 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.679930 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36798: BlockingRecv error: recv got EOF from 127.0.0.1:36798 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 15:50:40.702975 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.700742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.0.97.61:40395)
0929 15:50:40.701003 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.701009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.701432 (+ 423us) server_negotiation.cc:407] Connection header received
0929 15:50:40.701652 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.701656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.701741 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.701838 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.702720 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36808: BlockingRecv error: recv got EOF from 127.0.0.1:36808 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260929 15:50:40.742827 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.740477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.0.97.61:40395)
0929 15:50:40.740724 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.740731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.741253 (+ 522us) server_negotiation.cc:407] Connection header received
0929 15:50:40.741438 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.741446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.741582 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.741709 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.742562 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36820: BlockingRecv error: recv got EOF from 127.0.0.1:36820 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 15:50:40.812657 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.810438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.0.97.61:40395)
0929 15:50:40.810675 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.810683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.811086 (+ 403us) server_negotiation.cc:407] Connection header received
0929 15:50:40.811356 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.811364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.811483 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.811620 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.812402 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36830: BlockingRecv error: recv got EOF from 127.0.0.1:36830 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 15:50:40.946873 23435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:50:40.944542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.0.97.61:40395)
0929 15:50:40.944791 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:40.944797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:40.945092 (+ 295us) server_negotiation.cc:407] Connection header received
0929 15:50:40.945246 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:40.945250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:40.945328 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:40.945427 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:50:40.946548 (+ 1121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36832: BlockingRecv error: recv got EOF from 127.0.0.1:36832 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:50:40.975651 23448 catalog_manager.cc:1259] Loaded cluster ID: 5812ccc4c9ac47eb8cc99ee55cd0e77c
I20260929 15:50:40.975836 23448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: loading cluster ID for follower catalog manager: success
I20260929 15:50:40.978487 23448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: acquiring CA information for follower catalog manager: success
I20260929 15:50:40.980563 23448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aaaf2859b15e489fb97d685712accda0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:50:41.022760 23462 catalog_manager.cc:1259] Loaded cluster ID: 5812ccc4c9ac47eb8cc99ee55cd0e77c
I20260929 15:50:41.022877 23462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: loading cluster ID for follower catalog manager: success
I20260929 15:50:41.025527 23462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: acquiring CA information for follower catalog manager: success
I20260929 15:50:41.027644 23462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce6d87cd608642e58f1ecf0e3dad2fc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:50:41.215080 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:41.207215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.97.61:40395)
0929 15:50:41.207493 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:41.207502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:41.207708 (+ 206us) server_negotiation.cc:407] Connection header received
0929 15:50:41.207854 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:41.207858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:41.208038 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:41.208194 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:41.209069 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.209869 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.211597 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.212055 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.212243 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:41.213103 (+ 860us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:41.214184 (+ 1081us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:41.214326 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:41.214576 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 15:50:41.214686 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 15:50:41.230166 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:41.222029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.0.97.60:35439)
0929 15:50:41.222721 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:41.222729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:41.222903 (+ 174us) server_negotiation.cc:407] Connection header received
0929 15:50:41.223394 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:41.223400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:41.223614 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:41.223838 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:41.224352 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.225180 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.226985 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.227484 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.227655 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:41.228342 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:41.229161 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:41.229456 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:41.229698 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 15:50:41.229816 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":331,"threads_started":1}
I20260929 15:50:41.245406 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:41.236941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.0.97.62:41023)
0929 15:50:41.237528 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:41.237538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:41.238303 (+ 765us) server_negotiation.cc:407] Connection header received
0929 15:50:41.238645 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:41.238654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:41.238896 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:41.239077 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:41.239969 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.241056 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.242872 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.243327 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.243475 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:41.244093 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:41.244653 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:41.244771 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:41.244951 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 15:50:41.245051 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":421,"threads_started":1}
I20260929 15:50:41.246614 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:41.237291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.0.97.61:40395)
0929 15:50:41.237627 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:41.237637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:41.238515 (+ 878us) server_negotiation.cc:407] Connection header received
0929 15:50:41.238789 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:41.238794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:41.238894 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:41.238971 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:41.240097 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.241264 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.244373 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.245031 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.245152 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:41.245796 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:41.245937 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:41.246007 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:41.246215 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 15:50:41.246312 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 15:50:41.247278 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:50:41.238846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.0.97.60:35439)
0929 15:50:41.239097 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:50:41.239106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:50:41.240091 (+ 985us) server_negotiation.cc:407] Connection header received
0929 15:50:41.240307 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:50:41.240312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:50:41.240421 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:50:41.240528 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:50:41.241078 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.242538 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.245007 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:50:41.245677 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:50:41.245782 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:50:41.246552 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:50:41.246656 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:50:41.246776 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:50:41.246930 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 15:50:41.247012 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:50:41.253273 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 23213
I20260929 15:50:41.254693 23275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:41.382417 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 23213
I20260929 15:50:41.405782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 23287
I20260929 15:50:41.407269 23349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:41.529831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 23287
I20260929 15:50:41.554152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQOquZz/build/asan/bin/kudu pid 23362
I20260929 15:50:41.555642 23424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:50:41.680574 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQOquZz/build/asan/bin/kudu with pid 23362
2026-09-29T15:50:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5084 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10149 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (395864 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST